milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,237 Commits
151 Branches
122 Tags
587 MiB
Commit Graph

1174 Commits (22b2b7dbbe0e34376100c7dbd008f231392ecbb0)

Author SHA1 Message Date
sunby ab21115d54 Init rand.Seed 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 15:25:55 +08:00
XuanYang-cn 403212c5fa Fix describe_index test cases 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-08 15:00:08 +08:00
FluorineDog ec0407d5f1 Support schema verifier 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-08 12:41:46 +08:00
sunby aeb3038545 Remove GetCount in data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 10:49:44 +08:00
ThreadDao 7a3223dc81 tags collection stats 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-03-08 10:09:48 +08:00
neza2017 3ef18e86f0 Add 0331 tag for test_partition 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-06 17:47:11 +08:00
neza2017 efa7306ef6 Remove with prefix 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-06 16:00:41 +08:00
zhenshan.cao ce5d7c7ecc Change retry times and sleep duration 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-06 11:52:43 +08:00
xige-16 f6a4bacbeb Delete field fieldIDs from ReleaseSegmentRequest’s photo 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-05 20:47:43 +08:00
dragondriver c68a8d35bf Remove timeout controlled by server, use timeout client passed instead 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-05 20:43:19 +08:00
neza2017 2835bcdc4e Update masterservice unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-05 20:41:34 +08:00
Xiangyu Wang e5d595564a Add log for singlenode 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-05 18:16:50 +08:00
yukun 33d8e13f96 Fix load-collection in singlenode 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-05 18:14:59 +08:00
quicksilver 74154a11a4 Fix deploy error during regression stage 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-05 16:52:45 +08:00
XuanYang-cn a5c6f40c60 Enable tests in test_index.py 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-05 15:21:33 +08:00
BossZou 1efcf605e2 Enlarge Proxy node max size of reveivable grpc message 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-03-05 11:16:23 +08:00
sunby ddddd65d10 Delete internal/errors package 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-05 10:15:27 +08:00
bigsheeper 447a15207e Add zap log to query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-05 09:21:35 +08:00
dragondriver 801396e3ef Disable register link logic, use proxy node directly 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-04 22:27:12 +08:00
xige-16 ebaa5d4f5b Fix error of repeated loading partition failures 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-04 19:21:44 +08:00
sunby 6dc938e6f4 Refactor code 
1. add error as return value
2. check assertion success

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-04 18:58:16 +08:00
FluorineDog ef98dab2a9 Support segcore config 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-04 17:09:48 +08:00
sunby dec0b40a66 Add log panic in log module 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-04 16:01:30 +08:00
yukun 36bae90635 Fix singlenode search timeout caused by selectcase in rmq_msgstream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-04 15:53:17 +08:00
neza2017 ba21b11e9e Fix masterservice :drop index 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-04 14:54:16 +08:00
xige-16 6cdd4d02fc Refactor seek function 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-04 14:49:51 +08:00
godchen 3ba0dc2f44 Change doc 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-04 10:35:28 +08:00
ophunter233 4a921ffd7e Stop listen to dd channel in query node 
Signed-off-by: ophunter233 <chengming.li@zilliz.com>
 2021-03-03 16:06:02 +08:00
sunby ae432436db Init start position when opening new segment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-02 15:52:42 +08:00
XuanYang-cn c0edffc63f Fix datanode update segment statistics bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-02 15:47:37 +08:00
xige-16 795cb74c85 Fix message stream miss ack 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-02 13:15:49 +08:00
FluorineDog b7a00e30e2 Fix memory leak 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-01 11:14:51 +08:00
bigsheeper 456d0e8c23 Refactor time tick align strategy 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-27 18:33:29 +08:00
FluorineDog 66146223ca Support flat 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-02-27 12:46:37 +08:00
zhenshan.cao 966cf42b54 Enable indexnode to build index asynchronously 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-27 10:45:03 +08:00
bigsheeper d3db2b0d82 Add zap log in query service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-27 10:11:52 +08:00
BossZou 7893ebdb40 Fix test insert case 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-02-26 17:59:13 +08:00
godchen 7b9fdd7f29 Add opentracing 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-26 17:44:24 +08:00
xige-16 50a5f97421 Delete componentStateResponse in queryService.proto 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-26 15:26:21 +08:00
cai.zhang f5977a1302 Change retry times and add retry logic 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-26 15:17:47 +08:00
neza2017 4ea27335fd Ignore index param from create_collection 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-26 14:40:53 +08:00
zhenshan.cao 007e7ea858 Add tso unittest and EnableMaxLogic 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-26 13:05:52 +08:00
zhenshan.cao 12ea873216 Fix bug: drop empty index should be considered as legal operation 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-26 11:19:10 +08:00
bigsheeper 9edbff0a6b Fix fields check 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-26 11:15:44 +08:00
XuanYang-cn d34e3e77d1 Fix typo 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-26 11:07:25 +08:00
XuanYang-cn 10374f15d7 Update log in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-26 10:13:36 +08:00
sunby e95cb78ec8 Add LoadCollection from master 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-26 09:23:39 +08:00
neza2017 c36c532c70 Remove unused test code 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-25 19:12:40 +08:00
xige-16 650bf1d993 Refactor release collection warn message 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-25 18:26:11 +08:00
FluorineDog bf75c2fbb4 Fix bug (#1053) 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-02-25 17:56:43 +08:00
Xiangyu Wang 38266b8569 Lock channel at produce and consume 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-02-25 17:44:29 +08:00
Cai Yudong 84110d2684 Add tests/benchmark and tests/python_test using new python SDK 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-02-25 17:35:36 +08:00
neza2017 d5631b09e8 Describe collection by id 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-25 16:08:56 +08:00
bigsheeper 7cb28428ce Prevent consuming after loading collection 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-25 15:08:50 +08:00
neza2017 4c3486c770 Fix timestamp allocator 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-25 14:31:38 +08:00
yukun a391d4fbbf Fix stuck problem of create_index 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-25 12:03:35 +08:00
cai.zhang 1d899513dd Unify service name in config file 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-24 18:46:19 +08:00
cai.zhang 9be4a107f9 Fix bug for free memory 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-24 18:25:34 +08:00
zhenshan.cao bbc65e0b14 Remove checking of VectorFieldMetricType in create_collection 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-24 18:14:59 +08:00
sunby c149995bcd Change return value of GetCollectionStatistics. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-24 17:56:03 +08:00
bigsheeper 08a020798e Release collection and partitions 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-24 17:24:51 +08:00
zhenshan.cao 0679954922 Refactor tso and global id allocator 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-24 17:12:06 +08:00
neza2017 e396c3f4ad Add zap log 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-24 16:25:40 +08:00
xige-16 4c491471ee Add release collection and release partition interface for query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-24 15:58:55 +08:00
BossZou b02a9d6b43 Enlarge rocksmq stream buffer size 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-02-24 15:16:09 +08:00
XuanYang-cn fd562f9f9c Update doc: add proxy graph 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-24 09:48:17 +08:00
zhenshan.cao 7c5e2d89f6 Read node port from config 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-23 18:08:17 +08:00
zhenshan.cao 6b392cbe58 Fix memory leak 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-23 14:35:00 +08:00
xige-16 d5379a82a8 Fix load binlog error when segment not flushed 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-23 14:13:33 +08:00
dragondriver b506754364 Add workflow of drop index in index service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-23 11:57:18 +08:00
zhenshan.cao bbfcbbdd68 Fix ci: paramstable and startup logic refactor 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-23 11:40:30 +08:00
xige-16 7a7a73e89c Fix high memory usage in pulsarTtStream 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-23 11:40:12 +08:00
FluorineDog 15dd17488e Support benchmark 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-02-23 10:47:21 +08:00
sunby ea1d9ea99a Replace log in dataservice 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-23 09:58:06 +08:00
dragondriver 68518fec38 Fix memory leak in indexnode 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-22 18:33:40 +08:00
neza2017 f0c16a228e Fix show segments 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-22 16:40:26 +08:00
yukun e786ad6aa8 Fix inconsistency of produce msg and consume msg with rmq 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-22 16:34:15 +08:00
sunby b07a371f25 Create a goroutine to receive msg for every consumer 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-22 09:58:34 +08:00
xige-16 861576f77a Checkout field ids when load segment in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-22 10:44:38 +08:00
BossZou 7ac1821cf1 Add RocksMQ throughout test 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-02-20 18:30:37 +08:00
BossZou ae0e8b1a19 Init data service param before new DataServive 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-02-20 16:45:49 +08:00
cai.zhang 5c8747cf40 Make proxy retry time more frequently 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-20 16:14:10 +08:00
yukun 346b0a9549 Add unittest for RmqTtMsgStream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-20 15:58:19 +08:00
neza2017 1d3fdebcf4 Add drop index for proxy 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-20 15:38:44 +08:00
cai.zhang 4c5a898e9a Add masterservice docker 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-20 11:15:19 +08:00
dragondriver 6ebb03791f Fix drop index interface 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-20 10:37:11 +08:00
zhenshan.cao b3891ca199 Correct docker file 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-20 10:15:37 +08:00
xige-16 f62078c027 Fix crash error when search 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-20 10:14:03 +08:00
xige-16 a6442cb600 Fix querynode memory leak 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-20 09:20:51 +08:00
sunby 5769c000a1 Find failed python tests and add skip mark "r0.3-test" 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-19 15:37:04 +08:00
bigsheeper d8f41db823 Fix search crash error 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-19 13:56:01 +08:00
sunby 5b61182140 Add zap log 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-18 19:53:55 +08:00
zhenshan.cao d404d87633 Fix bug of getindexstate 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-19 09:52:06 +08:00
xige-16 450ea631bc Fix queryService assign queryChannel failure 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-18 16:26:02 +08:00
neza2017 549bda93ea Fix get index by name 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-18 15:58:59 +08:00
groot af3736aff6 Fix rocksmq bug 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-18 14:13:55 +08:00
neza2017 85c1404393 Refactor master:index ids 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-11 08:41:59 +08:00
xige-16 70fe5233cf Watch queryChannel on query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-09 17:09:26 +08:00
groot 833f1d49ab Let singlenode use rocksmq 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-09 15:57:10 +08:00
neza2017 e2d8358cb4 Refactor master:index ids 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-09 13:11:55 +08:00
neza2017 c2956dbe92 Change master meta prefix 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-09 10:36:31 +08:00
zhenshan.cao 7901d98d5e Fix bug: return wrong IndexID 0 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-08 18:42:33 +08:00
sunby c74221e4da Change ttMsgStream of dataservice to MsgStream 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-08 16:42:56 +08:00
neza2017 4035083188 Support Index name 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-08 14:49:12 +08:00
sunby 90d69658d9 Fix master meta table bugs 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-08 15:19:48 +08:00
groot dc3736281a Integrate message stream 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-08 14:30:54 +08:00
zhenshan.cao faa23fde86 Fix getindexstate failed to fetch indexstate 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-08 14:20:29 +08:00
dragondriver da94fbed0a Parse to map when the key of index params is named params 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-07 21:32:37 +08:00
bigsheeper 12b2eaf196 Rename apis 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-07 21:26:03 +08:00
zhenshan.cao 1ba8e2448f Add tsLoop for indexservice 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-07 18:38:58 +08:00
bigsheeper a64c831ed9 Fix segment replacement bug 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-07 18:37:18 +08:00
bigsheeper 7342e0758c Fix index and flush errors, and fix master crash error 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-07 17:02:13 +08:00
zhenshan.cao 2c9e226703 Add log for segment logic in proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-07 16:44:11 +08:00
xige-16 c7ac238aac Fix error when query node release collection 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-07 15:56:57 +08:00
xige-16 2ca53fa668 Fix msgstream deadlock when loadCollection 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-07 15:47:10 +08:00
yukun 563a8d66ae Add RmqTtMsgStream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-07 15:10:13 +08:00
sunby 6ee6557981 Record segment flushed state 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-07 14:51:52 +08:00
sunby eba5b9ae27 Fix queryservice client init bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-07 13:53:40 +08:00
bigsheeper b07b2484dc Fix error when loading 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-07 10:29:58 +08:00
dragondriver 8f4995dab5 Fix DropCollection task in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-07 09:56:39 +08:00
xige-16 cdcf006690 Fix loadCollection error when query service assign insert channels and query nodes 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-07 09:30:48 +08:00
zhenshan.cao 39161cac9a Add collectonScheam when calling load_collection and load_partition 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-06 21:17:18 +08:00
xige-16 a12b184e02 Add field schema to LoadCollection and LoadPartition photo 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-06 17:37:10 +08:00
bigsheeper e685f1825b Fix unittest caused by param table 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-06 15:16:10 +08:00
groot b034740c6c Make paramstable initialize once by using do_once 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-06 13:39:15 +08:00
yukun 099220394e Add rmq_msgstream unittest and fix bugs 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-06 11:39:12 +08:00
bigsheeper e9ee9a273e Refactor load service and check insertion binLog periodically 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-06 11:35:35 +08:00
sunby 0111cba8ae Fix startup bugs in some modules 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-06 10:28:53 +08:00
XuanYang-cn dd72dfcbd1 Fix queryservice start 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-05 17:57:41 +08:00
godchen c231f6da79 Fix collectionName not found error 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-05 16:32:03 +08:00
xige-16 57ba236117 Add mutex for message stream’s asConsumer 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-05 15:42:53 +08:00
dragondriver 7e1887da12 Add start.sh and stop.sh, remove them later 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 14:31:40 +08:00
bigsheeper a7eae3a4c1 Remove consumer in query node's initialization 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-05 14:15:10 +08:00
dragondriver e6aec3fd08 Hardcode the peer id of proxy node 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 14:09:55 +08:00
dragondriver 6ef82a59e5 Fix the bug when close the message stream of InsertTask 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 13:49:51 +08:00
godchen 0f620ab149 Fix invalidate metacache error 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-05 11:49:13 +08:00
dragondriver e384222803 Fix the logic of node id allocator 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 11:25:52 +08:00
neza2017 bea1535425 Add call of release collection when drop collection 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-05 11:05:37 +08:00
bigsheeper 1578c13224 Refactor collection replica 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-05 10:53:11 +08:00
dragondriver a48590a2df Add query service logic to Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 10:29:14 +08:00
bigsheeper 2ee54541aa Fix query node address 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-05 10:25:47 +08:00
sunby 9c3ea487f6 Fix close insert msg stream bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-05 10:19:52 +08:00
dragondriver 75e549da4e Change sdk version used in regression to 0.0.19 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 10:09:03 +08:00
XuanYang-cn a64c4efde9 Refactor collectionReplica in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 20:31:23 +08:00
sunby be596123d1 Fix dataservice startup 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-04 20:03:19 +08:00
godchen c2ca2c276f Fix response check error 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-04 19:34:35 +08:00
cai.zhang 4f2a1e7912 Fix bug for dataservice dockerfile 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-04 18:11:19 +08:00
dragondriver e0e8e1605e Implement the release-related task and load-related task in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-04 17:47:19 +08:00
sunby 7270d18c98 Fix timesync startup bugs 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-04 17:31:02 +08:00
xige-16 aaac839075 Joint debug query service success with other components 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-04 17:09:18 +08:00
XuanYang-cn 560d2350ae Fix datanode watchDmChannel bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 16:09:34 +08:00
godchen d6200a5196 Fix meta cache error 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-04 15:54:16 +08:00
cai.zhang 55634dc5f3 Add interface of load and release collection and partition 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-04 15:31:02 +08:00
yukun e3fadc45d4 Fix for new msgstream interface 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-04 15:23:21 +08:00
zhenshan.cao b89e5a3240 Add GetQuerySegmentInfo 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-04 14:37:12 +08:00
sunby 71b6c88c6e Defer lock's unlock in meta::DropSegment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-04 11:52:10 +08:00
Xiangyu Wang 8d35e76ad1 Change Msgstream interface 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-02-04 13:55:47 +08:00
bigsheeper c6c99ef32b Implement getSegmengInfo function 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-04 11:40:14 +08:00
XuanYang-cn 1aafe86ffe Change MsgPosition Logic 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 11:19:48 +08:00
dragondriver b92ff69cea Implement GetIndexState in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-04 11:08:36 +08:00
XuanYang-cn bc7ad02df4 Fix datanode init bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 10:25:01 +08:00
godchen 6e7e0b748a Send timetick to insert channel 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-03 20:04:29 +08:00
sunby e7588d3250 Implement getSegmentInfo interfaces 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-03 19:45:52 +08:00
neza2017 1189332786 Create default partition when create collection 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-03 18:41:20 +08:00
zhenshan.cao 1b616fb874 Add persistent_segmentinfo 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-03 18:55:00 +08:00
bigsheeper f12a0490bb Do not drop collections or partitions in flow graph 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 18:12:48 +08:00
Xiangyu Wang fd9e42d3ed Update msgstream/unmarshal interface 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-02-03 17:30:10 +08:00
bigsheeper abffdbff62 Remove partition tag 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 16:44:38 +08:00
dragondriver 6c3f169ecc Add collection id and partition id to insert & search workflow 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-03 16:40:23 +08:00
zhenshan.cao b853020ef9 Remove redudent ctx and openTracing logic 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-03 16:34:56 +08:00
yukun 35f6f4b7c2 Add consume in rmq_msgstream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-03 15:18:54 +08:00
XuanYang-cn 73a8e7de84 Fix datanode insertchannel refresh bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-03 15:18:05 +08:00
sunby 208700b0e7 Set fixed insert channel number, remove channel manager 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-03 15:07:48 +08:00
bigsheeper 01e9dc8e3f Remove collection name 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 11:52:19 +08:00
sunby f3aad3a71c Change SegmentInfo 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 18:53:10 +08:00
bigsheeper 5e781b9370 Remove field name in query node and segCore 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 10:10:07 +08:00
sunby 45b99c0cf3 Save indexName and indexID in IndexCodec 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 19:56:11 +08:00
neza2017 899d31e72d Change interface: add index id and index name params 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-02 20:09:46 +08:00
zhenshan.cao bfa14d40a8 Rename indexID to indexBuildID and add IndexID and IndexName 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-02 19:56:04 +08:00
godchen 66d5189601 Add get collection statistics implemetation 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-02 19:54:31 +08:00
Xiangyu Wang 32c5a92449 Update interface in msgstream 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-02-02 16:32:15 +08:00
dragondriver e4e3ec88d1 Add usage count of insert channels 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-02 15:49:13 +08:00
bigsheeper 09bdbee944 Refactor load service, remove seg manager 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-02 15:13:08 +08:00
sunby ca9c2f5c12 Add GetCount and implement GetCollectionStatistics 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 14:25:58 +08:00
xige-16 4cd315408b Refactor query service’s meta interface 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-02 11:52:41 +08:00
sunby 46f14a5e69 Fix bugs in GetInsertChannels 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 11:14:17 +08:00
zhenshan.cao 3afa7dd50d Fix GetSegmentStates 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-02 10:58:39 +08:00
neza2017 5de89422ed Support multi index 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-02 10:09:10 +08:00
XuanYang-cn 90d471b133 Refactor main of queryservice and querynode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-02 09:52:42 +08:00
godchen 93fd6b70f3 Fix the error usage of ttmsg 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-01 19:22:35 +08:00
sunby a74767fe57 Fix bugs 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-01 16:20:13 +08:00
neza2017 3f45cfd6e8 Fix data node 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-01 11:44:02 +08:00
godchen 94dfab5b21 Check collection whether exists before drop 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-01 10:53:13 +08:00
godchen abcc565311 Add proxy metacache 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-31 14:55:36 +08:00
zhenshan.cao c2914dd113 Refactor segment allocator, use collectionID and partitionID 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-31 10:15:54 +08:00
neza2017 854accf95b Remove master and writenode 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-30 16:41:47 +08:00
bigsheeper 3573e01093 Check index periodically and load index, add indexing segment type 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-30 16:02:10 +08:00
dragondriver 996111bf8c Refactor insert channels used in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-30 15:30:38 +08:00
sunby 3a67dda06c Add NewDataService in main 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-30 11:49:48 +08:00
neza2017 51eac0eb84 Use GetCollectionByName instead 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-30 11:38:02 +08:00
dragondriver 454c05f2de Implement the InvalidateCollectionMetaCacheTask in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-30 10:40:22 +08:00
dragondriver 4d5a9e067c Fix the cancel logic of proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-29 19:25:32 +08:00
zhenshan.cao bfb6766f7c Refactor indexservice 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-29 18:12:21 +08:00
dragondriver f660caca0c Refactor the client interface of proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-29 17:29:31 +08:00
Xiangyu Wang 99cef4b5c8 Add Distributed main entrypoint 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-01-29 17:08:31 +08:00
bigsheeper 73d2b6a101 Get index param from minio and filter by vector fields 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-29 15:22:24 +08:00
neza2017 bdf84f08ab Add watcher_test and fix some bugs 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-29 14:40:19 +08:00
sunby f088467dde Add watcher_test and fix some bugs 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-28 10:14:31 +08:00
XuanYang-cn b597a637f3 Add datanode metatable tests 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-29 12:03:10 +08:00
neza2017 305c814215 Fix master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 19:37:25 +08:00
zhenshan.cao ee0faddb07 Refactor proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-29 09:27:26 +08:00
dragondriver 7957170220 Dock Proxy with other service component 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-28 20:51:44 +08:00
neza2017 3a866dab3d Update proto 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-28 17:25:43 +08:00
neza2017 2ebeade25e Fix get component states 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-28 17:13:00 +08:00
sunby 2be8cc1c4b Add index params serde to IndexCodec 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-28 16:41:24 +08:00
sunby bcd7368533 Fix get component states 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-28 15:20:24 +08:00
sunby 5ac463c9ca Add datanode client in data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-28 11:24:41 +08:00
XuanYang-cn 4acaa97039 Add unittests for datanode and queryservice 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-27 20:00:47 +08:00
sunby f1afd5d311 Add loop to update tsoallocator and id allocator in master service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-27 16:38:18 +08:00
bigsheeper 68cd15af63 Get index info from master 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-27 14:41:56 +08:00
dragondriver a0226c09f0 Change the logic node id allocator in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-27 13:52:01 +08:00
sunby b4bcaaf847 Fix data service startup bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-27 11:40:10 +08:00
XuanYang-cn 05b013c52a Fix datanode bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-27 11:37:26 +08:00
xige-16 947976dbec Add test for query service’s load function 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-27 11:34:16 +08:00
sunby dbfe68512d Add dbID to data service proto 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-27 10:33:17 +08:00
sunby a3adb22e10 Change params 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-27 09:59:17 +08:00
bigsheeper 4f914a2c30 Add docker file and main function of query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-27 09:50:52 +08:00
neza2017 4d37a329da Fix data service init 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 19:25:17 +08:00
neza2017 d1b23cb8c5 Add debug info 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 19:24:09 +08:00
sunby f632ed0272 Fix data service init 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 18:45:12 +08:00
zhenshan.cao e83ac41bca Add component for index 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-26 18:01:32 +08:00
neza2017 8c54af292e Add dd handler 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 17:55:15 +08:00
neza2017 c4f4ae8627 Fix master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 17:47:38 +08:00
XuanYang-cn 0b1c4f0420 Add building datanode cmd in Makefile 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-26 17:40:26 +08:00
sunby 9f72633fd9 Add dd handler 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 17:06:13 +08:00
neza2017 2ea6a5f9ee Add make dataservice in Makefile 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 16:41:10 +08:00
sunby b4997a37a2 Add make dataservice in Makefile 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 16:36:26 +08:00
neza2017 b6a788373b Fix proxy server 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 16:30:36 +08:00
neza2017 fd0227b69a Add logic of get time tick 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 15:23:42 +08:00
sunby 3b08162059 Add main function 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 15:14:49 +08:00
xige-16 7c210310dc Add load function for query service 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-26 15:13:20 +08:00
dragondriver ad244dc4e8 Add initial interface to proxy service client 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-26 15:09:31 +08:00
dragondriver d972b75aaa Add GetTimeTickChannel, GetComponentStates to proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-26 14:55:57 +08:00
XuanYang-cn b445587e11 Add datanode main 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-26 14:46:54 +08:00
cai.zhang 346bb1ba49 Delete some unused code and some debug logs 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-26 14:31:31 +08:00
dragondriver c6a6b1436c Add soft time tick in proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-26 13:41:41 +08:00
XuanYang-cn 6c303c67b2 Rearrange update segment statistics of start position procedure 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-26 09:44:39 +08:00
neza2017 5af23cf018 Init params and add segment start/end position 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 09:43:41 +08:00
cai.zhang f940cc455a Add dockerfile for index 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-26 09:38:40 +08:00
neza2017 7f3aa92d10 Add docker file for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 15:58:04 +08:00
FluorineDog a8fd6e7686 Enable drop index/field_data and extract info visitor 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-25 18:59:44 +08:00
yukun 5600d06583 Add global_rmq 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-25 18:53:08 +08:00
bigsheeper 7554246ace Add segment seeking and use real client 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-25 18:33:10 +08:00
sunby 7c0c835b49 Init params and add segment start/end position 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-25 15:16:28 +08:00
XuanYang-cn 5015a1035d Add logic of get meta from MasterService for datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-25 17:10:32 +08:00
neza2017 1fe099775d Refactor data service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 15:17:17 +08:00
neza2017 e89e89b894 Add main for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 12:03:44 +08:00
sunby 54fcd486bb Refactor data service. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-25 13:45:56 +08:00
XuanYang-cn 5d9d35b733 Add Supplementary statistics 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-25 13:36:40 +08:00
FluorineDog 69f6963d45 Enable lock and fix row count 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-25 11:20:25 +08:00
XuanYang-cn 9fe672ab86 Enchance datanode interface 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-24 21:20:11 +08:00
neza2017 5aec8bc5e6 Refactor masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-24 20:26:35 +08:00
bigsheeper 445675a588 Refactor loadSegment request and load strategy 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-24 18:02:08 +08:00
zhenshan.cao aef4e41e91 Enable proxynode microbatch pulsar msg and enable parallel test 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-23 20:58:46 +08:00
sunby 82d6fb18b8 Add init params returned to data node 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-23 20:22:59 +08:00
XuanYang-cn b794921b7e Add enum Role in internal.proto 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-23 18:56:08 +08:00
neza2017 e2b45f9866 Init param table and open unittest at ci 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-23 17:56:57 +08:00
sunby c23e2de435 Implment data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-23 14:41:29 +08:00
sunby 2f7319cdbb Add interfaces with other modules 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-22 19:43:27 +08:00
neza2017 92e3c519d9 Use FlushCompletedMsg instead 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-23 10:40:34 +08:00
BossZou 01267776d2 Fix rocksmq consume invalid currendID 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-01-23 10:15:57 +08:00
neza2017 e68d50f29e Invalid collection meta caceh && ingore time stamp if not dd request 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-23 10:12:41 +08:00
XuanYang-cn 50e2369000 Change name and tag to ID 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-22 19:36:09 +08:00
FluorineDog bff208d78c Add sealedSegment (go&c) unittest, fix growingSegment field id check 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-22 17:40:58 +08:00
bigsheeper 067c30c422 Add sealedSegment cgo unittest, fix growingSegment field id check 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-22 15:41:54 +08:00
xige-16 511aa419fa Refactor query service 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-22 14:28:06 +08:00
neza2017 2d2dbd7d19 Add logic of segment flush complete 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-22 11:06:12 +08:00
dragondriver cd52adc18b Add proxy service to ci workflow 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-22 12:57:23 +08:00
neza2017 90a0b8b2f5 Update proto 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-22 11:44:02 +08:00
bigsheeper 46a0a6133a Implement distributed api, delete partition states api 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-22 11:17:18 +08:00
sunby ef79b8d8c7 Add client and grpc service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-22 11:07:07 +08:00
XuanYang-cn 377ee8687d Modify flush logic 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-22 09:36:40 +08:00
dragondriver eba6bb288c Refactor the sructure of proto file 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-22 09:36:18 +08:00
neza2017 9828a52999 Add create index 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-21 10:01:29 +08:00
FluorineDog 7f044fff82 Enable search for sealed 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-21 15:29:52 +08:00
sunby be6775671b Add error returned by Init,Start,Stop functions 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-19 17:38:01 +08:00
cai.zhang 83c60fd84f Update proto about index 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-21 15:20:23 +08:00
XuanYang-cn ec27788628 Add segment statistics updates logic for datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-21 09:55:25 +08:00
cai.zhang 7619d968fd Generate random node port for indexService 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-20 18:26:20 +08:00
xige-16 76bc365181 Add seek function for pulsarTtMsgStream 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-20 17:34:50 +08:00
FluorineDog 5e06dc1732 Enable segment sealed, add support for loadXXX 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-20 17:33:31 +08:00
yukun 527c0c49df Add rocksmq unittest with goroutines 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-20 16:46:58 +08:00
cai.zhang c35079d7e7 Update registerNode in indexservice 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-20 10:15:43 +08:00
BossZou 4588342fa1 Add IdAlocator service 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-01-20 15:02:23 +08:00
sunby 2af61fc196 Change ComponentStates in proto 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-20 11:08:04 +08:00
sunby 3602376a10 Rename GetServiceStates to GetComponentStates and reconstruct NodeStates and ServiceStates 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-20 11:02:29 +08:00
Xiangyu Wang 7ee8623e1a Refactor msgstream 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-01-20 10:02:59 +08:00
zhenshan.cao 2ec573d2f9 Recover delete docs 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-20 09:36:50 +08:00
cai.zhang fb493cc235 Update interface name of GetIndexStates 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-19 18:32:57 +08:00
neza2017 918e279869 Set msg stream 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-19 16:14:13 +08:00
sunby 7e8162a79e Add dd handler 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-19 15:35:40 +08:00
neza2017 d5e53aa28a Add Meta table implementation 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-19 14:13:49 +08:00
bigsheeper 2064b014a1 Add loadFieldData, use partition id instead of tag 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-19 14:44:03 +08:00
sunby 76a7684d96 Add data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-19 12:10:49 +08:00
XuanYang-cn 6a13386393 Add DataNode package 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-19 11:37:16 +08:00
neza2017 1ce32d8775 Add master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-19 09:45:42 +08:00
bigsheeper aaf8b0ad40 Add sealed segment, segment type and manager, send loadIndexReq to service's buffer 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-19 10:29:16 +08:00
yukun b874a55c35 Add complex unittest for rocksmq 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-19 10:22:27 +08:00
zhenshan.cao 76d92e73d1 Replace old proto with new proto 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-18 21:39:56 +08:00
xige-16 d22c2cd203 Add proto for query service loading collection 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-18 19:32:08 +08:00
yukun 3aca35b10c Add rocksmq unittest 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-18 15:05:49 +08:00
FluorineDog 92261e38c5 Add system property and optimize easy assert 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-18 13:56:20 +08:00
bigsheeper 48821690b3 Rename Service interface to Component, and update segmentLoad strategy 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-18 10:38:41 +08:00
GuoRentong dd6c4bfb75 Update doc:interface definitions related to LoadCollection 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2021-01-18 10:09:17 +08:00
FluorineDog 9ff1bc0f99 Adapt Expr Executor to SegmentInternalInterface, fix row_count consistency 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-16 18:12:58 +08:00
xige-16 0164e78bb5 Add query service module 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-16 15:31:10 +08:00
XuanYang-cn 37688559f1 Fix multi rowdata writenode crash bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-16 15:06:19 +08:00
bigsheeper 46a14da3eb Implement segment manager and loadSegment 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-16 15:04:32 +08:00
FluorineDog 16ff16b212 Renaming and add load index api beta 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-16 10:48:01 +08:00
bigsheeper f02bd8c8f5 Rename query node package, implement watchDmChannel 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-16 10:12:14 +08:00
yukun 44c75bdd2a Add consume function in rocksmq 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-15 20:12:26 +08:00
FluorineDog c0a3a509f7 Remove todos, implement chunk_data and chunk_scalar_index 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-15 18:23:50 +08:00
sunby 67e6ea8051 Add data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-15 17:09:41 +08:00
dragondriver e6225a7030 Add service interface to milvus.proto 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-15 16:51:21 +08:00
BossZou 31178d6a5c Add procude logic of RocksMQ 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-01-15 16:06:35 +08:00
bigsheeper b98b226d8c Refactor query node and implement channelRemove, channelAdd 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-15 15:28:54 +08:00
godchen 0971cf65b2 Remove opentracing stdlog 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-15 15:28:27 +08:00
quicksilver 7fe061b846 Raneme init_devcontainer.sh to devcontainer.sh 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-01-15 14:38:36 +08:00
neza2017 56ab2bda2d Update proto:add partition id 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-15 11:53:22 +08:00
zhenshan.cao 63b21321d6 Refactor indexservice and update doc 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-15 13:50:27 +08:00
FluorineDog 8fccf7e630 Use bitset_view at search 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-15 13:40:51 +08:00
FluorineDog 7d81222550 Support span in SegmentGrowing, refine vector_trait 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-15 10:32:50 +08:00
zhenshan.cao 04e2062750 Update proto 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-14 20:30:27 +08:00
XuanYang-cn fb63983eb2 Modify rocksdb compilation 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-14 15:12:08 +08:00
yukun 6e36f10a26 Fix rocksdb illegal instruction in some machine 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-14 14:24:14 +08:00
FluorineDog 63ca5f8031 Add span and segment interface, rename segment_growing and use interface as base 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-13 18:46:25 +08:00
Xiangyu Wang 52bd6abdf8 Change RocksMQ interface 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-01-13 17:26:57 +08:00
cai.zhang 3d885742ee Reopen ruleguard and format code 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-13 13:36:27 +08:00
GuoRentong 1104f059ee Update doc:module interfaces 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2021-01-13 11:08:03 +08:00
quicksilver 8e9d8e36e1 Update run_go_unittest.sh 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-01-13 10:40:46 +08:00
FluorineDog 4cd42c553f Rename field_name, make field_id strongly typed, skip multithread test 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-12 18:31:52 +08:00
bigsheeper 2d2af1505a Improve mutex usage in query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-12 18:03:24 +08:00
GuoRentong f3d2afa4f4 Update doc:detailed interface definition 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2021-01-11 18:35:54 +08:00
bigsheeper 01781103a9 Fix incorrect usage of msgStream and illegal check in master 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-11 18:02:22 +08:00
quicksilver a6690dbc99 Fix syntax error: Bad fd number 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-01-11 15:25:19 +08:00
yukun 18cfd31d40 Fix Load and Remove with prefix 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-11 15:17:06 +08:00
dragondriver 9920877e51 Refactor retry logic with max sleep time 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-11 11:07:01 +08:00
bigsheeper 4fad3b189c Fix search without insertion, improve nil hits behavior (#622) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-09 17:58:18 +08:00
xige-16 004d0027b3 Fix error that different loadIndex messages have same indexParam 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-09 15:57:55 +08:00
zhenshan.cao 02df310fa5 Fix data race 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-09 12:30:22 +08:00
xige-16 f4566731fc Fix search error in regression test 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-09 09:47:22 +08:00
sunby 5dfe9448ae Modify test_index 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-08 16:00:16 +08:00
yukun d11f00cd9f Add rocksdb in kv 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-08 19:07:57 +08:00
GuoRentong 82ad488555 Update doc: rocksmq 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2021-01-08 18:37:24 +08:00
dragondriver 018466a256 Avoid crash when index parameters are invalid 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-08 17:45:57 +08:00
XuanYang-cn 65089ea362 Fix flush sync test bug and remove unecessary lock in meta_table 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-08 17:20:59 +08:00
GuoRentong c4599e26de Update doc: node api 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2021-01-08 16:59:19 +08:00
bigsheeper e13822043d Fix search test when partition not exists, add binary vector search validation 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-08 16:41:27 +08:00
godchen fa73f17f94 Add opentracing doc 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-08 15:32:38 +08:00
quicksilver f6d4214bbe Optimize the workflow for cache go mod 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-01-08 15:00:58 +08:00
sunby 3b68bf22f3 Add pre-checks in create index task 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-08 11:06:56 +08:00
dragondriver 92b2df14ca Fix unsupported index combinations 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-08 14:10:10 +08:00
zhenshan.cao 6a3ec57fef Fix docker lib dependency 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-07 21:08:23 +08:00
dragondriver a4860beae2 Parse type params and index params when buildindex is invoked 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-07 17:38:46 +08:00
sunby 6ffe8739bf Add unit tests 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-07 17:24:40 +08:00
xige-16 d9d2f33a23 Add binary test for loadIndex and fix loadIndexService can't close correctly 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-07 17:22:10 +08:00
XuanYang-cn e42f8bca84 Fix dd flush not completed err 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-07 16:59:05 +08:00
cai.zhang f5ea46b1a3 Simplify proto and fix logic bug of checking vector type 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-07 16:13:28 +08:00
sunby 3ae04794f6 Fix bug in describeIndexProgressTask 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-07 15:39:20 +08:00
FluorineDog 34a00aacfa Enable binary index in segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-07 15:00:42 +08:00
dragondriver ced56c84f8 Add support to build binary vector index 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-07 14:56:17 +08:00
godchen 8f1f118b26 Add query opentracing 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-07 14:24:35 +08:00
sunby c1a758cb58 Fix bug of segment manager 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-07 14:05:08 +08:00
XuanYang-cn b187bb289d Fix writenode flush sync test 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-07 11:47:39 +08:00
FluorineDog 3bf205d9a8 Fix inner product 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-07 09:32:17 +08:00
sunby bf3a6dc3c6 Remove collection in segment manager when dropping collection 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-06 19:23:31 +08:00
xige-16 8715cd1f0c Fix key error when loading index 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-06 18:19:44 +08:00
bigsheeper a1e2ff3c26 Add gcNode to write node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-06 16:44:12 +08:00
neza2017 4015d7245d Merge operation 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-06 14:45:50 +08:00
bigsheeper 810be533ab Add gcNode, and remove msg's downstream index 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-06 14:02:25 +08:00
cai.zhang 1cfc2ff0a5 Add make clean in Makefile 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-06 13:50:48 +08:00
sunby da6eeddbf8 Add flush timestamp 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-06 13:14:38 +08:00
FluorineDog 5a26f6ef21 Enable sub_query_result 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-06 12:01:13 +08:00
zhenshan.cao b06e01f523 Initmate querynode collection manager 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-06 11:31:36 +08:00
godchen 1c87faf8b3 Add insert request opentracing 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-06 11:17:35 +08:00
FluorineDog e16a7478da Add verifier 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-05 20:13:54 +08:00
dragondriver bccf8d2098 Add query support for ivf-flat, nsg 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-05 19:20:55 +08:00
sunby d0c787628b Pass type params when building index 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-05 16:09:14 +08:00
bigsheeper e9dc96f931 Fix bucket name in index builder and query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-05 15:58:13 +08:00
XuanYang-cn 05f1350e19 Add writenode to ci 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-05 15:14:59 +08:00
quicksilver ca89ece576 Fix go format bug 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-01-05 14:20:45 +08:00
neza2017 8b39643e44 Fix write node crashed if collection meta not existed 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-05 12:20:46 +08:00
XuanYang-cn fa77783ad7 Fix print ddl binlog bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-05 12:20:30 +08:00
zhenshan.cao 9f339ef8ce Fix bug: indexbuilder req not initialized 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-05 11:57:27 +08:00
sunby a1151d9bf5 Change kvRootPath to MetaRootPath in master 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-05 11:31:42 +08:00
sunby 95b162ccfd Refactor flush scheduler 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-05 10:57:59 +08:00
bigsheeper d5d9fa03ea Disable the dropCollection filtering 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-04 20:46:00 +08:00
bigsheeper 33e1e17045 Fix writer client DescribeSegment api 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-04 20:16:15 +08:00
sunby 0a383608f3 Remove mock client in master 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-31 11:20:03 +08:00
bigsheeper 248be309fd Add index builder unites 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-04 17:55:01 +08:00
FluorineDog 89f38e459a Fix bool type 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-04 17:52:42 +08:00
zhenshan.cao 6bd5702441 Add indexbuilder address to paramstable 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-04 14:53:20 +08:00
bigsheeper cb3ca1ab7c Refactor kvPath param and unify the segment flush kvPath in write node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-04 14:17:59 +08:00
XuanYang-cn 03b5d32569 Add default fields when create collection 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-04 12:03:29 +08:00
zhenshan.cao 7ed63c79ce Remove system field of describle_collection in proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-04 11:41:19 +08:00
cai.zhang 7a28cfb9e7 Fix bug of sdk : unnecessary is_primary_key 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-04 10:58:51 +08:00
dragondriver e61903ea36 Add query interface for cgo index wrapper 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-04 10:42:51 +08:00
bigsheeper 1ba47ac433 Check system field in segCore 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-04 10:13:01 +08:00
bigsheeper b7fc945aca Add flush msg unmarshal function register 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-31 17:25:45 +08:00
xige-16 bfb18df7e7 Parse result_id from target entry 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-31 16:53:57 +08:00
quicksilver 1927eaf88b Add revive linter 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-31 16:38:37 +08:00
zhenshan.cao 59878b1dbb Add lazy connect logic for indexbuilder 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-31 15:44:06 +08:00
dragondriver 6cd8c2528d Add parameter check for almost all index type 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-31 15:16:03 +08:00
xige-16 29b21b4bf2 Refactor reduce strategy in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-31 14:49:27 +08:00
sunby 4808b46246 Filter indexable fields in flush scheduler 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-30 19:56:54 +08:00
dragondriver 2e198e2783 Add support for idmap, nsg, sq8 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-30 19:37:45 +08:00
FluorineDog 09620efadf Merge from master 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-30 18:50:56 +08:00
zhenshan.cao adf77c423c Update knowhere 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-30 18:30:51 +08:00
bigsheeper 4835d82060 Add index builder test cases 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-30 18:16:34 +08:00
sunby e7998da797 Fix bug when starting indexbuilder 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-30 16:41:24 +08:00
XuanYang-cn 3a2e2a04b6 Fix writenode bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-29 21:15:00 +08:00
bigsheeper bb8da15691 Return status and add try catch exception in index builder 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-29 17:53:27 +08:00
quicksilver b1b8747477 Add logic of incremental check for go code 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-29 17:19:44 +08:00
dragondriver 243346247b Add unittest case which used in regression to index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-29 16:31:03 +08:00
bigsheeper 1e3b7cd55e Fix binary vector metric type validation 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-29 16:14:52 +08:00
xige-16 be32a33eff Refactor loadIndexTest to test diffrent index params 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-29 14:43:40 +08:00
XuanYang-cn 884f66d56d Fix timesync start twice in master 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-29 14:10:22 +08:00
cai.zhang f80536ed77 Update the logic of running go unittest 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-29 11:56:37 +08:00
XuanYang-cn 9577da9fce Add writenode main 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-28 18:52:34 +08:00
FluorineDog eb8717188e Enable load index 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-28 17:55:48 +08:00
dragondriver f67aae9596 Fix the codec bug of cgo wrapper of index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-28 16:55:50 +08:00
bigsheeper 5b9e8c4f60 Add write node client segment flushing 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-28 15:25:16 +08:00
godchen d489183798 Random generate querynode channel in unittest 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-28 15:24:20 +08:00
XuanYang-cn aab6dede4a Add Flush for DDBuffer and insertBuffer 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-28 11:29:17 +08:00
dragondriver 11cef6e978 Add support to build binary index 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-28 10:00:02 +08:00
GuoRentong e8ec0424d8 Update doc: service api 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2020-12-27 09:05:24 +08:00
neza2017 7e8b203fe2 Add time tick in wite node 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-26 17:05:24 +08:00
XuanYang-cn e316533a3a Add Flush Sync service 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-26 16:28:55 +08:00
bigsheeper 5acc9681d6 Fix search error about metric type 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-26 15:29:48 +08:00
xige-16 d599407e2b Add unittest for loadIndex service 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-26 14:16:51 +08:00
cai.zhang 2031c54746 Add unittest about index in proxy 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-26 11:51:07 +08:00
sunby a56f8f7b9e Add runtime stats in master 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-25 18:32:03 +08:00
sunby e21397d68a Add runtime stats unittest 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-25 16:08:35 +08:00
sunby a03ab05c6c Add DescribeIndex and DescribeIndexProgress RPC 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-25 15:39:56 +08:00
dragondriver 84a0cf7097 Add unittest to c index wrapper 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-25 11:10:31 +08:00
XuanYang-cn c0797333ce Add ddl binlog metaTable in writenode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-24 20:55:40 +08:00
neza2017 4dbf4ade41 Add test for master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-24 16:19:42 +08:00
zhenshan.cao a0244b7683 Add segment expire logic for proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-24 16:53:31 +08:00
bigsheeper 4ecdea698f Refactor query node and query serviceMain changes:1. Add ddBuffer and save binLog. 
2. Trans to insertData.
3. Change dataFields data to Data, dim to Dim4. Add float vector and binary vector.5. Deserialize data and convert to InsertData.6. Move all data into InsertData.7. Add insert buffer and hash string.
8. Add minIOkV in insertBuffer node.
9. Init write node insertBuffer maxSize from writeNode.yaml.
10. Add ddBuffer.
11. Add ddBuffer binLog and minio.
12. Add ddNode unittest.
13. Remove redundant call.
14. Increase test time.
15. Delete ddl const, use request's timestamp instead.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-24 15:38:29 +08:00
xige-16 f5198bcaee Add LoadIndexService in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-24 14:51:55 +08:00
sunby add31188ba Add CreateIndex RPC 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-24 14:50:16 +08:00
neza2017 0afeaa78c8 Fix unittest of flow graph 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-24 14:10:00 +08:00
sunby 5d58606349 Move Status to SegIDAssignment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-24 10:11:34 +08:00
cai.zhang a8b78f11c3 Update proto about index 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-23 19:06:11 +08:00
FluorineDog 342f4cb741 Support dynamic chunk size 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-23 19:02:37 +08:00
XuanYang-cn 7ce0f27ebc Add buffer to minIO for binlogs 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-23 18:06:04 +08:00
sunby e7ebfcb05a Save index meta to meta table 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-23 15:13:45 +08:00
godchen e52b130ba6 Refactor data codec code 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-23 15:46:16 +08:00
sunby 0b9cffaf87 Remove index_name in IndexRequest and remove indexType in FieldIndexMeta 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-23 11:34:35 +08:00
zhenshan.cao 11316525e5 Fix bug of running unittest in parallel 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-22 22:08:03 +08:00
sunby 2aa483c922 Add index meta to meta table 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-22 20:16:32 +08:00
dragondriver adfb5187cf Dock build index task with storage & indexbuilder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-22 00:14:36 +00:00
cai.zhang 05918f55d2 Impl CreateIndex interface for proxy server 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-22 15:39:10 +08:00
sunby 3a1c210098 Add runtime stats to record field stats 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-21 20:24:40 +08:00
godchen 8f87ee632e Add field id when create collection 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-21 19:31:24 +08:00
godchen e90ef211c2 Add flush, build index, load index scheduler 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-21 19:30:24 +08:00
cai.zhang b582cf7b8e Add the interface of index in proto 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-21 19:28:54 +08:00
dragondriver d023c1afba Refactor index wrapper, use protobuf instead of json to pass data 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-21 16:36:07 +08:00
XuanYang-cn 87521adfbd Add logic of deserilize rawData into InsertData 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-21 16:27:03 +08:00
FluorineDog d5daa18392 Enable most DSL-related cases 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-21 14:45:00 +08:00
sunby 06620935ab Move expire timestamp to SegIDAssignment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-21 10:51:37 +08:00
zhenshan.cao 9694203f8b Add writer client and metatable for writer 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-21 10:06:46 +08:00
sunby f360ed7004 Add unittests 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-21 09:56:35 +08:00
sunby c71cd40f68 Refactor segment manager 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-19 12:55:24 +08:00
FluorineDog e65cfe1e3d Enable complex dsl parser 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-19 10:36:49 +08:00
zhenshan.cao 0d75840ed6 Fix missing file 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-18 18:35:03 +08:00
neza2017 0110ba6bd2 List events of binlog 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-12 16:52:09 +08:00
dragondriver da9ea7d2b9 Add cgo wrapper to index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-18 15:44:27 +08:00
godchen 475d36a2c9 Add data codec 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-18 15:21:25 +08:00
zhenshan.cao 3f6e926aef Refactor indexbuilder 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-13 06:48:05 +08:00
FluorineDog 1540cffbb1 Add executor for expression 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-12 18:17:53 +08:00
cai.zhang 54f2b79f1b Add indexbuilder client 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-12 17:10:42 +08:00
neza2017 07d68de2a3 Fix data codec 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-11 18:14:19 +08:00
bigsheeper 585d3f9831 Refactor param table, and add ddNode 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-11 17:20:14 +08:00
neza2017 8df05714a8 Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-11 12:01:38 +08:00
FluorineDog e54d5ccf19 Fix push failed because git hook detects "wrong error" 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-11 14:59:13 +08:00
XuanYang-cn d64aa37db3 Add KFlush msg type 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-11 12:01:20 +08:00
zhenshan.cao ca34353e9b Add writernode client 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-11 11:38:32 +08:00
xige-16 0067e2d989 Add load index client in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-11 11:29:54 +08:00
cai.zhang 62e9002348 Add git-hooks for check code 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-11 11:29:07 +08:00
FluorineDog 02f855359a Fix term bug and add todo note 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-10 20:13:37 +08:00
zhenshan.cao fc054874d3 Add index_builder 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-10 17:55:55 +08:00
bigsheeper 762b1e1f1f Add ddService, fieldID, requestID and refactor params table, fix nil hits search result bug 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-10 16:31:09 +08:00
neza2017 9585819154 Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-10 15:50:09 +08:00
neza2017 9cbebc0221 Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-10 14:52:42 +08:00
sunby 16071d92b5 Modify metatable close segment interface. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-10 13:35:09 +08:00
quicksilver 21721d2e03 Cache go mod workflow for jenkins pipeline 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-10 09:57:14 +08:00
neza2017 5512ce8d8a Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-10 09:31:29 +08:00
XuanYang-cn 881be7f3e3 Impl minimum runable writenode and writenode flowgraph 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-09 20:07:27 +08:00
neza2017 de12fa5a10 Add binlog event 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-09 11:18:13 +08:00