milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,238 Commits
136 Branches
115 Tags
568 MiB
Commit Graph

512 Commits (3a866dab3d059ef926dac50c05e09393e48f923a)

Author SHA1 Message Date
dragondriver 6cd8c2528d Add parameter check for almost all index type 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-31 15:16:03 +08:00
xige-16 29b21b4bf2 Refactor reduce strategy in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-31 14:49:27 +08:00
sunby 4808b46246 Filter indexable fields in flush scheduler 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-30 19:56:54 +08:00
dragondriver 2e198e2783 Add support for idmap, nsg, sq8 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-30 19:37:45 +08:00
FluorineDog 09620efadf Merge from master 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-30 18:50:56 +08:00
zhenshan.cao adf77c423c Update knowhere 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-30 18:30:51 +08:00
bigsheeper 4835d82060 Add index builder test cases 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-30 18:16:34 +08:00
sunby e7998da797 Fix bug when starting indexbuilder 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-30 16:41:24 +08:00
XuanYang-cn 3a2e2a04b6 Fix writenode bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-29 21:15:00 +08:00
bigsheeper bb8da15691 Return status and add try catch exception in index builder 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-29 17:53:27 +08:00
quicksilver b1b8747477 Add logic of incremental check for go code 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-29 17:19:44 +08:00
dragondriver 243346247b Add unittest case which used in regression to index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-29 16:31:03 +08:00
bigsheeper 1e3b7cd55e Fix binary vector metric type validation 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-29 16:14:52 +08:00
xige-16 be32a33eff Refactor loadIndexTest to test diffrent index params 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-29 14:43:40 +08:00
XuanYang-cn 884f66d56d Fix timesync start twice in master 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-29 14:10:22 +08:00
cai.zhang f80536ed77 Update the logic of running go unittest 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-29 11:56:37 +08:00
XuanYang-cn 9577da9fce Add writenode main 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-28 18:52:34 +08:00
FluorineDog eb8717188e Enable load index 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-28 17:55:48 +08:00
dragondriver f67aae9596 Fix the codec bug of cgo wrapper of index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-28 16:55:50 +08:00
bigsheeper 5b9e8c4f60 Add write node client segment flushing 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-28 15:25:16 +08:00
godchen d489183798 Random generate querynode channel in unittest 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-28 15:24:20 +08:00
XuanYang-cn aab6dede4a Add Flush for DDBuffer and insertBuffer 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-28 11:29:17 +08:00
dragondriver 11cef6e978 Add support to build binary index 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-28 10:00:02 +08:00
GuoRentong e8ec0424d8 Update doc: service api 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2020-12-27 09:05:24 +08:00
neza2017 7e8b203fe2 Add time tick in wite node 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-26 17:05:24 +08:00
XuanYang-cn e316533a3a Add Flush Sync service 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-26 16:28:55 +08:00
bigsheeper 5acc9681d6 Fix search error about metric type 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-26 15:29:48 +08:00
xige-16 d599407e2b Add unittest for loadIndex service 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-26 14:16:51 +08:00
cai.zhang 2031c54746 Add unittest about index in proxy 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-26 11:51:07 +08:00
sunby a56f8f7b9e Add runtime stats in master 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-25 18:32:03 +08:00
sunby e21397d68a Add runtime stats unittest 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-25 16:08:35 +08:00
sunby a03ab05c6c Add DescribeIndex and DescribeIndexProgress RPC 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-25 15:39:56 +08:00
dragondriver 84a0cf7097 Add unittest to c index wrapper 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-25 11:10:31 +08:00
XuanYang-cn c0797333ce Add ddl binlog metaTable in writenode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-24 20:55:40 +08:00
neza2017 4dbf4ade41 Add test for master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-24 16:19:42 +08:00
zhenshan.cao a0244b7683 Add segment expire logic for proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-24 16:53:31 +08:00
bigsheeper 4ecdea698f Refactor query node and query serviceMain changes:1. Add ddBuffer and save binLog. 
2. Trans to insertData.
3. Change dataFields data to Data, dim to Dim4. Add float vector and binary vector.5. Deserialize data and convert to InsertData.6. Move all data into InsertData.7. Add insert buffer and hash string.
8. Add minIOkV in insertBuffer node.
9. Init write node insertBuffer maxSize from writeNode.yaml.
10. Add ddBuffer.
11. Add ddBuffer binLog and minio.
12. Add ddNode unittest.
13. Remove redundant call.
14. Increase test time.
15. Delete ddl const, use request's timestamp instead.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-24 15:38:29 +08:00
xige-16 f5198bcaee Add LoadIndexService in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-24 14:51:55 +08:00
sunby add31188ba Add CreateIndex RPC 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-24 14:50:16 +08:00
neza2017 0afeaa78c8 Fix unittest of flow graph 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-24 14:10:00 +08:00
sunby 5d58606349 Move Status to SegIDAssignment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-24 10:11:34 +08:00
cai.zhang a8b78f11c3 Update proto about index 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-23 19:06:11 +08:00
FluorineDog 342f4cb741 Support dynamic chunk size 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-23 19:02:37 +08:00
XuanYang-cn 7ce0f27ebc Add buffer to minIO for binlogs 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-23 18:06:04 +08:00
sunby e7ebfcb05a Save index meta to meta table 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-23 15:13:45 +08:00
godchen e52b130ba6 Refactor data codec code 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-23 15:46:16 +08:00
sunby 0b9cffaf87 Remove index_name in IndexRequest and remove indexType in FieldIndexMeta 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-23 11:34:35 +08:00
zhenshan.cao 11316525e5 Fix bug of running unittest in parallel 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-22 22:08:03 +08:00
sunby 2aa483c922 Add index meta to meta table 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-22 20:16:32 +08:00
dragondriver adfb5187cf Dock build index task with storage & indexbuilder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-22 00:14:36 +00:00
cai.zhang 05918f55d2 Impl CreateIndex interface for proxy server 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-22 15:39:10 +08:00
sunby 3a1c210098 Add runtime stats to record field stats 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-21 20:24:40 +08:00
godchen 8f87ee632e Add field id when create collection 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-21 19:31:24 +08:00
godchen e90ef211c2 Add flush, build index, load index scheduler 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-21 19:30:24 +08:00
cai.zhang b582cf7b8e Add the interface of index in proto 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-21 19:28:54 +08:00
dragondriver d023c1afba Refactor index wrapper, use protobuf instead of json to pass data 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-21 16:36:07 +08:00
XuanYang-cn 87521adfbd Add logic of deserilize rawData into InsertData 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-21 16:27:03 +08:00
FluorineDog d5daa18392 Enable most DSL-related cases 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-21 14:45:00 +08:00
sunby 06620935ab Move expire timestamp to SegIDAssignment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-21 10:51:37 +08:00
zhenshan.cao 9694203f8b Add writer client and metatable for writer 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-21 10:06:46 +08:00
sunby f360ed7004 Add unittests 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-21 09:56:35 +08:00
sunby c71cd40f68 Refactor segment manager 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-19 12:55:24 +08:00
FluorineDog e65cfe1e3d Enable complex dsl parser 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-19 10:36:49 +08:00
zhenshan.cao 0d75840ed6 Fix missing file 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-18 18:35:03 +08:00
neza2017 0110ba6bd2 List events of binlog 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-12 16:52:09 +08:00
dragondriver da9ea7d2b9 Add cgo wrapper to index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-18 15:44:27 +08:00
godchen 475d36a2c9 Add data codec 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-18 15:21:25 +08:00
zhenshan.cao 3f6e926aef Refactor indexbuilder 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-13 06:48:05 +08:00
FluorineDog 1540cffbb1 Add executor for expression 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-12 18:17:53 +08:00
cai.zhang 54f2b79f1b Add indexbuilder client 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-12 17:10:42 +08:00
neza2017 07d68de2a3 Fix data codec 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-11 18:14:19 +08:00
bigsheeper 585d3f9831 Refactor param table, and add ddNode 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-11 17:20:14 +08:00
neza2017 8df05714a8 Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-11 12:01:38 +08:00
FluorineDog e54d5ccf19 Fix push failed because git hook detects "wrong error" 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-11 14:59:13 +08:00
XuanYang-cn d64aa37db3 Add KFlush msg type 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-11 12:01:20 +08:00
zhenshan.cao ca34353e9b Add writernode client 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-11 11:38:32 +08:00
xige-16 0067e2d989 Add load index client in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-11 11:29:54 +08:00
cai.zhang 62e9002348 Add git-hooks for check code 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-11 11:29:07 +08:00
FluorineDog 02f855359a Fix term bug and add todo note 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-10 20:13:37 +08:00
zhenshan.cao fc054874d3 Add index_builder 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-10 17:55:55 +08:00
bigsheeper 762b1e1f1f Add ddService, fieldID, requestID and refactor params table, fix nil hits search result bug 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-10 16:31:09 +08:00
neza2017 9585819154 Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-10 15:50:09 +08:00
neza2017 9cbebc0221 Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-10 14:52:42 +08:00
sunby 16071d92b5 Modify metatable close segment interface. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-10 13:35:09 +08:00
quicksilver 21721d2e03 Cache go mod workflow for jenkins pipeline 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-10 09:57:14 +08:00
neza2017 5512ce8d8a Add binlog unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-10 09:31:29 +08:00
XuanYang-cn 881be7f3e3 Impl minimum runable writenode and writenode flowgraph 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-09 20:07:27 +08:00
neza2017 de12fa5a10 Add binlog event 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-09 11:18:13 +08:00
FluorineDog 968ddc6016 Fix thrift.a not found 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-09 14:57:07 +08:00
sunby ed54d3e26c Fix binlog reader bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-09 09:55:56 +08:00
quicksilver d09ad77fb2 Add archiving file workflow for cache thirdparty 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-08 19:53:22 +08:00
sunby 8ace1c3837 Add binlog serialization and deserialization 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-08 19:13:39 +08:00
XuanYang-cn e6f726e73a Add cache for thirdparty files cache 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-08 18:51:07 +08:00
XuanYang-cn 32977e270c Add impl cgo of parquet 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-12-08 14:41:04 +08:00
zhenshan.cao 65ce1f97e7 Fix bug: pulsar dirty msg 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-07 19:41:39 +08:00
zhenshan.cao 7bbbc14637 Fix bug: address already used 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-07 15:22:20 +08:00
xige-16 37d7526d31 Fix search test failure 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-07 14:37:42 +08:00
neza2017 0d8273c7cc Add string type payload cgo wrapper 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-06 15:15:11 +08:00
neza2017 aeca8f85a5 Run cwrapper unittest in ci 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-05 17:39:58 +08:00
cai.zhang 806a97505c Add query node for CI progress 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-05 16:12:45 +08:00
neza2017 70710dee47 Add parquet payload 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-05 16:11:03 +08:00
FluorineDog 63c8f60c6e Enable term parser and executor 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-05 09:51:27 +08:00
FluorineDog 6412ebc0d4 Add support of metric type in schema, enable binary vector, fix segfault 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-05 06:46:01 +08:00
xige-16 acdf8cc9e1 Fix memory leak in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-04 19:32:30 +08:00
cai.zhang 18a9ffb617 Increase timeout for golangci-lint 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-04 17:57:24 +08:00
zhenshan.cao 0b6fe0a8cf Refactor paramtable 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-04 14:18:17 +08:00
godchen 7ab5b5d80d Add MinIO kv implements 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-12-03 19:00:11 +08:00
neza2017 cec903da19 Add parquet payload 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-02 18:31:56 +08:00
xige-16 36cf8a8ea7 Support return primary key when doing search 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-12-03 17:18:06 +08:00
quicksilver 0530fdf62f Open regression test 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-03 14:05:52 +08:00
bigsheeper 707a6a366e Fix issue about collection cannot be found 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-12-02 17:13:16 +08:00
zhenshan.cao 569057251c Add support of custom ID 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-02 16:12:21 +08:00
zhenshan.cao 23aebe7e2a Fix proxy unittest 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-12-01 19:53:53 +08:00
neza2017 17cc7bc337 Use MetaRootPath instead 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-12-01 17:03:39 +08:00
dragondriver 84b15c1f5c Fix the bug of meta cache in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-01 16:41:25 +08:00
cai.zhang c1fbc4f8fa Open the tests for insert 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-12-01 15:18:57 +08:00
FluorineDog 6b74d82259 Support some validation for dsl parser 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-12-01 02:16:53 +08:00
zhenshan.cao fda8d62b38 Fix make error 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-30 22:14:19 +08:00
neza2017 d3858500c6 Add static check 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-30 19:38:23 +08:00
dragondriver 2358fae703 Check the status of search results before reduce in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-30 19:08:32 +08:00
bigsheeper 0cd3e8d86c Use exception instead of runtime error 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-30 17:58:23 +08:00
dragondriver 51a9f49d35 Fix the topk check when deal with the search result in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-30 17:46:00 +08:00
cai.zhang 546beb333d Compile master and proxy without cgo 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-30 16:40:13 +08:00
zhenshan.cao 24942c3d6d Fix insert failed to return correct id range 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-30 12:08:47 +08:00
bigsheeper d88a654db8 Fix search result 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-30 11:22:59 +08:00
quicksilver 86573d0053 Create Jenkins pipeline 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-30 10:07:04 +08:00
FluorineDog 77fa75b1ec Add binary insert and warper of binary search, rename vector 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-30 05:18:44 +08:00
cai.zhang 85a544b79b Add dockerfile for each component 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-28 19:06:48 +08:00
cai.zhang 115d0732af Update docker-compose file for unittest 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-28 16:43:39 +08:00
dragondriver eb6c0e0a28 Fix the bug when reduce the multiple search result from query nodes 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-28 15:57:49 +08:00
godchen 92bf09c7e6 Add ddstream and scheduler implements 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-11-28 15:26:39 +08:00
dragondriver 568cef0730 Fix InsertTask and SearchTask 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-28 10:48:29 +08:00
FluorineDog c9fb34142c Enable primary_key switch 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-28 09:16:00 +08:00
dragondriver 235e723eb6 Read proxy id from param table instead of hardcode 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-27 11:17:12 +08:00
XuanYang-cn 3e605ae25f Rename reader to querynode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-27 10:56:53 +08:00
bigsheeper 1decc1a4ca Add metaRootPath and kvRootPath 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-27 09:59:40 +08:00
neza2017 c507abdeaa Fix start time sync 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-26 16:51:44 +08:00
bigsheeper f21c0ef2e9 Reduce cpu usage 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-26 17:58:08 +08:00
dragondriver 72a7af3f4a Add time tick logic to Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-26 17:47:46 +08:00
neza2017 32445dd02b Add cpu profile 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-26 16:01:31 +08:00
cai.zhang 57da66ba72 Update read config logic in docker 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-26 15:18:36 +08:00
dragondriver ab75438abf Fix bug of repack function of InsertTask in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-26 15:17:20 +08:00
neza2017 d9e5a8774d Add dimension validation 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-26 15:07:47 +08:00
FluorineDog 8ae573b580 Fix bool bug 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-26 14:56:58 +08:00
sunby e8e21a0308 Add dimension validation. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-26 14:54:03 +08:00
zhenshan.cao 045fc3d808 Fix segment fault 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-26 14:29:01 +08:00
XuanYang-cn 0a16a9a651 Add logic of getting milvus distributed config from master by grpc 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-26 13:21:06 +08:00
dragondriver 8ce0deca4e Add default partition tag to InsertTask in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-26 11:38:04 +08:00
xige-16 ae78aa1ac1 Warp cpp reduce and repack searchResultMsg to specified pulsar channel 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-26 11:12:33 +08:00
neza2017 ae26a81550 Change to use Marshal as text 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-26 10:13:23 +08:00
sunby f4c643f1bd Add number limitation of partition and field. And Change default partition tag to "_default" 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-25 18:39:05 +08:00
zhenshan.cao f12366342f Fix bug and delete unused code 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-25 16:45:42 +08:00
bigsheeper 4ed11d9775 Reopen segment test assertion 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-25 16:24:57 +08:00
bigsheeper 3b24c52a8c Catch cpp runtime error 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-25 16:20:07 +08:00
dragondriver a7b3efecd7 Read query node num from param table instead of hardcode 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-25 16:17:33 +08:00
cai.zhang 4bcb460c98 Add clang-format 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-25 15:25:00 +08:00
FluorineDog a6f950e5b6 Fix boost::dynamic_bitset misuse 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-25 15:17:17 +08:00
sunby 24b29bec30 Add collectionName/partitionTag/fieldName validation. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-25 14:44:30 +08:00
FluorineDog a38f539b9b Move c headers to segcore folder 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-25 10:31:51 +08:00
FluorineDog df1dcf81b9 Update cpplint.py and lint all files 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-24 21:28:38 +08:00
xige-16 1a929ac655 Fix memory leak when running query node’s unites 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-24 20:14:51 +08:00
FluorineDog 83379977b2 Enable structured index 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-24 19:09:57 +08:00
quicksilver d1f31c7b4e Cache Docker volumes in code-checker.yaml 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-24 17:04:48 +08:00
bigsheeper 41fb2d2991 fix lock usage 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-24 16:12:39 +08:00
bigsheeper 8ec7e4576e Add dropPartition and createPartition 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-24 15:59:51 +08:00
neza2017 ad0078c07a Load grpc port of proxy from config 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-24 11:50:21 +08:00
cai.zhang a13638b172 Add cpplint and optimize the check of c++ code 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-23 19:57:05 +08:00
bigsheeper 84f3d974fa Delete pulsar address test and refactor master param table 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-23 18:04:09 +08:00
dragondriver c7a49c97cc Add repack function of InsertTask in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-23 16:52:17 +08:00
cai.zhang 71beaadaf4 Update README for script 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-23 16:40:42 +08:00
FluorineDog 76001b29b6 Support Range Query 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-23 11:56:45 +08:00
neza2017 3277987898 Add segment threshold factor 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-23 11:41:53 +08:00
sunby 578e952c60 Add segment threshold factor 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-23 10:20:30 +08:00
zhenshan.cao 7c39fc24a9 Simplify proxy config 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-21 19:12:59 +08:00
sunby 2ebf3b55e4 Modify pulsar address in unittest 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-21 18:36:14 +08:00
sunby 3afab6ecb9 Add unittest and fix a bug in segment manager 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-21 17:11:05 +08:00
quicksilver 859ff62bc9 Unify Network for UnitTest 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-21 16:54:20 +08:00
cai.zhang 8ff82c2fd5 Add judgement when executing clang-format 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-21 16:37:48 +08:00
dragondriver e36b565ec0 Normalize the naming of variable and function in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-21 16:31:38 +08:00
FluorineDog 9f35dee2df Support range executor 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-21 15:56:13 +08:00
neza2017 73e3ff09f5 Refactor variable name in Master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-21 15:53:01 +08:00
XuanYang-cn 03c996d71a Fix timesync bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-21 15:25:23 +08:00
cai.zhang 17c3a70d64 update ParameterTable 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-21 15:06:46 +08:00
GuoRentong 8b88a9c306 Update docs: master::syncMsgProducer 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2020-11-21 11:39:23 +08:00
dragondriver aabe9afd75 Add meta cache in Proxy and use meta cache to generate id in InsertTask 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-20 17:53:31 +08:00
neza2017 2514e8b8a9 Optimize code in Master module 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-20 17:46:49 +08:00
bigsheeper dd3654f953 Fix query node param table 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-20 17:10:24 +08:00
quicksilver 1728a4f419 Update .devcontainer.json 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-20 15:02:26 +08:00
neza2017 cda47a9252 Remove unused code 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-20 09:13:29 +08:00
zhenshan.cao 50d59d91ee Add segmentIDAssigner for proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-19 21:02:31 +08:00
dragondriver 0b2add0d7b Fix query result loop logic in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-19 18:45:24 +08:00
neza2017 429c94e18b Add grpc for proxy 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-19 17:09:22 +08:00
xige-16 cf11212932 Refactor reduce using cgo in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-19 14:13:39 +08:00
neza2017 ce89f64bb3 Set master config 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-19 10:46:17 +08:00
neza2017 c3de667bf5 Implement segment management in master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-18 19:46:18 +08:00
bigsheeper 99ea7e9f05 Update param table 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-18 19:33:56 +08:00
sunby 9e4ce8454e Implement segment management in master 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-18 15:04:17 +08:00
FluorineDog e7dd30a884 Add framework of ExecExprVisitor 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-18 17:32:52 +08:00
quicksilver 057673edc9 Fix permission denied in gdbserver container 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-18 16:38:28 +08:00
dragondriver a18ac17a98 Increase timeout interval of task in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-18 15:44:17 +08:00
neza2017 544c701f98 Start timeSyncMsgProducer in master 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-18 10:14:11 +08:00
cai.zhang c7861c2dba Add the check of C++ code in the CI progress 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-18 10:07:05 +08:00
dragondriver b32e55d5f0 Add timeout logic to task in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-17 20:00:23 +08:00
quicksilver 72ed65d7f8 Check builder images 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-17 18:36:03 +08:00
godchen 4f04be4a28 Add master dd task unittest 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-11-17 17:17:10 +08:00
bigsheeper e117a6097a Fix pulsar url 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-17 16:32:34 +08:00
xige-16 4e8efe85df Delete wrong useage of *TsMsg 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-17 14:10:07 +08:00
XuanYang-cn 14ad7dc4a2 Add ccache and optimize index unittest compilation 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-17 11:05:15 +08:00
xige-16 833fee59d9 Add searchService for query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-17 10:07:42 +08:00
cai.zhang 5bcabffdaf Replace conf struct with ParamsTable 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-16 21:10:43 +08:00
dragondriver aff1e79f82 Fix SearchTask in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-16 17:01:10 +08:00
FluorineDog 8c48cf30c0 Format and and extra parameters for checking 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-16 15:41:56 +08:00
neza2017 9d3a21a9c9 Rewrite NewTSOKVBase 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-16 12:37:46 +08:00
xige-16 d59f6ac6ca Add queryNodeSegStatsMsg for msgStream 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-16 10:55:49 +08:00
dragondriver 23d9ddb4a3 Add unittest to grpc service that Proxy provide 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-14 18:18:10 +08:00
zhenshan.cao 9f948d3095 Remove mockmaster 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-14 16:40:10 +08:00
quicksilver 00b4b67a1f Update the version of build environment 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-14 16:01:12 +08:00
neza2017 e349508e78 Fix list-collections 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-14 15:57:39 +08:00
XuanYang-cn fd6d0c78e5 Refactor master scheduler: show_collections 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-14 15:37:07 +08:00
zhenshan.cao e94138816f Add tsLoop for master 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-14 15:26:14 +08:00
neza2017 ce969e9568 Update segment 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-14 14:39:52 +08:00
XuanYang-cn 27032cc126 Refactor master scheduler: describe_collection 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-14 13:06:41 +08:00
neza2017 74f8c45a23 Add ListCollection for meta table 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-14 11:50:57 +08:00
dragondriver c2382adf04 Initialize the task scheduler of Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-14 11:24:49 +08:00
FluorineDog f47fc7fef1 Using std::string instead of char * for PlaceholderGroup 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-13 19:37:59 +08:00
godchen 4a3ca1632b Add master partition task implementation 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-11-13 17:20:13 +08:00
zhenshan.cao 595d827155 Add grpc check logic for master 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-13 16:53:55 +08:00
bigsheeper 83e37d6e36 Add RWMutex to container, use container interface 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-13 16:47:47 +08:00
XuanYang-cn 1b48e84c35 Refactor master scheduler: has_collection 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-13 15:44:02 +08:00
cai.zhang 0b478b614a Change Id to ID in go files according to golint check 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-13 15:17:18 +08:00
sunby e541041b28 Add QueryNodeSegStats proto 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-11-13 14:20:51 +08:00
XuanYang-cn 88cb274644 Refactor master scheduler: drop_collection 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-13 12:46:24 +08:00
quicksilver 792f0679e5 Running as root is not permitted on build environment 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-11-13 12:27:41 +08:00
cai.zhang cfee642a11 Add ruleguard and open ruleguard check 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-13 10:00:32 +08:00
cai.zhang b68322d469 Open golangci-lint check 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-12 17:58:05 +08:00
neza2017 14d4a400dd Load etcd port from etcd 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-12 17:15:01 +08:00
XuanYang-cn 85db2c0438 Fix timesync tests 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-12 15:56:23 +08:00
bigsheeper 93bd786506 Update getPartitionByTag api with collectionName 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-12 14:46:02 +08:00
bigsheeper b059b02d7f Format id to ID 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-12 14:34:20 +08:00
FluorineDog f0ec7a8c78 Support Placeholder Parser and refine CreatePlan on predicate-less 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-12 12:18:07 +08:00
cai.zhang 1cd2fd9954 Add the standard of golangci-lint 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-12 12:04:12 +08:00
godchen 4a11a6431b Add time sync producer 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2020-11-12 11:21:21 +08:00
bigsheeper c9c8fb9def Close pulsar channel 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2020-11-12 11:18:23 +08:00
cai.zhang 3b2c86c42d Fix bug for cmake_build dir was changed 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-12 10:45:46 +08:00
xige-16 205abfbbcd Add pulsar message stream for system 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2020-11-11 20:14:20 +08:00
neza2017 d71083bf6b Change master scheduler: CreateCollection 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2020-11-11 20:01:04 +08:00
zhenshan.cao 170792d9bc Unify proto package and version 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-11 18:13:20 +08:00
zhenshan.cao 057563d936 Refactor master 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-11-11 17:53:54 +08:00
XuanYang-cn a2289d3ce6 Add softTimeTickBarrier and hardTimeTickBarrier 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2020-11-11 16:41:53 +08:00
dragondriver fb1e24ade8 Refactor task interface of Proxy and fix the wait logic of task 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-11 16:00:56 +08:00