milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,407 Commits
133 Branches
115 Tags
559 MiB
Commit Graph

680 Commits (a6a7dc9c8f9b5191d0eb3a141a604df3c33d0925)

Author SHA1 Message Date
bigsheeper 44622622f6
[skip ci] Add comment for getSegmentsMemSize (#9433) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-08 06:43:26 +08:00
bigsheeper 8ab25a1bb1
[skip ci] Add comment for deleteCollection (#9367) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-06 23:38:00 +08:00
bigsheeper c86193fb7b
[skip ci] Add comment for newCollection (#9366) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-06 23:36:13 +08:00
bigsheeper 0e4ee6d420
[skip ci] Add comment for getLoadType (#9365) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-06 23:34:24 +08:00
bigsheeper 12dcbe3d78
[skip ci] Add comment for setLoadType (#9299) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-05 22:44:00 +08:00
bigsheeper eab57af117
[skip ci] Add comment for checkReleasedPartitions (#9297) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-05 22:40:34 +08:00
bigsheeper 8b30d9aeb5
[skip ci] Add comment for deleteReleasedPartition (#9296) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-05 22:38:44 +08:00
zhenshan.cao d835adb442
[skip ci] Fix golint error in querynode (#9289) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-05 21:38:16 +08:00
zhenshan.cao a4cddd950b
[skip ci] Fix golint error in querynode (#9288) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-05 21:36:24 +08:00
congqixia 1891adb7e3
Add QueryNode implementation Assertion (#9259) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-05 12:42:04 +08:00
bigsheeper 040618187b
[skip ci] Add comment for addReleasedPartition (#9244) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-05 11:19:23 +08:00
bigsheeper 4427b24171
[skip ci] Add comment for setReleaseTime (#9242) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-05 11:17:29 +08:00
bigsheeper c1b9f4303f
[skip ci] Add comment for getReleaseTime (#9243) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-05 11:15:14 +08:00
bigsheeper be9ab19b7e
[skip ci] Add comment for getPChannels (#9174) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-04 08:22:51 +08:00
bigsheeper 91e4adff15
[skip ci] Add comment for addPChannels (#9173) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-04 08:18:59 +08:00
bigsheeper 358d2e1e07
[skip ci] Add comment for getVChannels (#9172) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-04 08:16:44 +08:00
zhenshan.cao 83b19a559f
[skip ci] Fix golint error in type_def.go (#9161) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-03 21:09:56 +08:00
bigsheeper 02b2f3c88b
[skip ci] Add comment for addVChannels (#9103) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-03 00:52:02 +08:00
bigsheeper 0b545efa4a
[skip ci] Add comment for removePartitionID (#9102) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-03 00:40:03 +08:00
bigsheeper 5b074732f5
[skip ci] Add comment for addPartitionID (#9101) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-03 00:38:13 +08:00
bigsheeper a9a4b89b0e
[skip ci] Add comment for Schema (#9045) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-01 23:44:33 +08:00
bigsheeper 66eac49c49
[skip ci] Add comment for HandleCProtoResult (#9044) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-01 23:44:10 +08:00
bigsheeper 4323b5dc98
[skip ci] Add comment for HandleCStatus (#9043) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-01 23:43:21 +08:00
bigsheeper a5a85f75bd
[skip ci] Add comment for newDataSyncService (#8962) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-01 09:53:43 +08:00
bigsheeper 4cf5843516
[skip ci] Add comment for newHistorical (#8963) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-10-01 09:45:12 +08:00
Xiaofan 2c2b800ff4
Change Log Name with NodeId (#8721) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-01 08:52:50 +08:00
bigsheeper e7d92e4e79
Add streaming segments memory to used memory when checkLoadMemory (#8940) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-30 21:54:08 +08:00
groot b17d253db1
Add unittest for distributed/querynode (#8834) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-30 10:02:02 +08:00
bigsheeper 6f2f09db3a
[skip ci] Add comment for MarshalForCGo (#8870) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-29 23:38:20 +08:00
bigsheeper 954e01aed8
[skip ci] Add comment for protoCGo.destruct (#8871) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-29 23:24:36 +08:00
dragondriver a10f421c14
Get SIMD type used in faiss (#8849) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-29 20:50:19 +08:00
zhenshan.cao d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-29 20:26:00 +08:00
congqixia db944cd09c
Refactor flowgraph and related invocation (#8770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-29 20:19:59 +08:00
dragondriver cf8600077f
Refactor the index file format (#8514) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-29 09:52:12 +08:00
bigsheeper 2417a5da69
[skip ci] Add comment for search in streaming (#8805) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-28 22:42:05 +08:00
bigsheeper 243ee06660
[skip ci] Add comment for collection id (#8806) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-28 22:40:05 +08:00
bigsheeper 2515452ea5
[skip ci] Add comment for search in historical (#8804) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-28 22:38:13 +08:00
bigsheeper 9d6c95bf25
Add cacheSize to prevent OOM in query node (#8765) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-28 22:24:03 +08:00
Xiaofan e0e7042b5e
Add test for tsafe (#8626) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-28 12:28:03 +08:00
bigsheeper f7eaff5145
[skip ci] Add comment for queryNodeFlowGraph (#8700) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-27 19:06:07 +08:00
bigsheeper 5490b84e77
[skip ci] Add comment for NewQueryNode func (#8699) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-27 19:02:09 +08:00
bigsheeper 692ceb32a3
[skip ci] Add comment for indexLoader (#8698) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-27 18:58:21 +08:00
Xiaofan 585989935a
Revert the logger change (#8677) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-27 17:37:57 +08:00
zhenshan.cao b71bffe26f
Replace ChannelID with ShardName in DML Request (#8539) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-27 10:01:59 +08:00
Xiaofan 8465d03dec
Make tsafe close not elegant (#8582) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:20:10 +08:00
Xiaofan d38adda8c2
Rename log file based on node id (#8517) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:18:11 +08:00
bigsheeper 057c7cb591
[skip ci] Add comment for historical (#8607) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-26 20:19:58 +08:00
bigsheeper 4b68be2d36
[skip ci] Add comment for addCollectionFlowGraph (#8606) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-26 20:18:03 +08:00
bigsheeper 78604cc1e1
[skip ci] Add comment for query node (#8605) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-26 20:16:12 +08:00
congqixia de3fd15aee
Fix tsafe close deadlock (#8567) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-26 14:29:57 +08:00
groot a20a9c71e2
Reduce go unittest time cost (#8556) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-26 12:56:07 +08:00
congqixia 6f7f71fdc0
Add lock before reading tSafeWatcher map (#8562) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-26 12:37:56 +08:00
congqixia c72f074009
Add waitgroup for tsafe close logic (#8545) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-26 11:07:56 +08:00
zhenshan.cao 2d9b4fedc9
[skip ci] Add comments for flow_graph_message.go (#8523) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-24 21:37:57 +08:00
bigsheeper e61bafc6f5
Init query node once (#8086) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-24 21:03:56 +08:00
bigsheeper d7f311a208
[skip ci] Add comment for streaming (#8498) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-24 20:54:01 +08:00
bigsheeper 977d55b66c
[skip ci] Add comment for task (#8497) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-24 20:53:49 +08:00
bigsheeper 6bc4379de5
[skip ci] Add comment for tSafe replica (#8495) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-24 20:52:13 +08:00
bigsheeper 52126f2d5a
Dynamic add tSafe watcher, use sync.Cond instead of selectCase, add ref count (#8050) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-24 13:57:54 +08:00
bigsheeper 1b7056d3c2
[skip ci] Add comment for dataSyncService in query node (#8418) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-23 21:48:12 +08:00
bigsheeper a6060e4481
[skip ci] Add comment for errors in query node (#8417) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-23 21:46:16 +08:00
bigsheeper 322f4ec80f
[skip ci] Add comment for type def in query node (#8415) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-23 21:44:17 +08:00
congqixia da6d93527d
Add session liveness check (#8408) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 18:29:55 +08:00
dragondriver b3f6bd2088
Expose simd type to query node metrics (#8299) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-23 11:27:54 +08:00
Xiaofan f5173b595f
Add a channel prefix for all channels (#8166) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-23 10:53:53 +08:00
bigsheeper 220d17500c
[skip ci] Add comment for query node flow graph (#8326) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-22 20:10:09 +08:00
bigsheeper 0e2b4a7cd8
Add comment for queryCoordMock (#8223) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-22 11:30:07 +08:00
bigsheeper 36dd794986
Add comment for type def in query node (#8220) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-22 11:28:04 +08:00
bigsheeper 1f1c738016
Add loadFieldData unittest for query node (#8219) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-18 18:32:07 +08:00
Xiangyu Wang c314b2b4a8
Remove unused label rules (#7929) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-18 18:19:51 +08:00
bigsheeper 11c65dc2b3
Add comment for collection replica (#8147) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-18 18:15:51 +08:00
bigsheeper 223044a7ca
Rename name of unittests in query node (#8225) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-18 18:05:51 +08:00
congqixia bdfb8768d4
Enable gosec (#8202) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-18 14:45:50 +08:00
Yusup ccb57ba4b2
add collection alias (#7255) 
Signed-off-by: Yusup <yusup@lsgrep.com>
 2021-09-18 11:13:51 +08:00
Cai Yudong 83bf9c5efb
Add knowhere.yaml to support knowhere configuration (#7967) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 10:37:51 +08:00
bigsheeper 338e32362b
Return error if setSegment failed when loadSegment (#8075) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-18 10:03:50 +08:00
Cai Yudong d10c6abc9e
Apply for msgstream from pool when creating collection (#7738) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 09:13:50 +08:00
bigsheeper ee3ab7a783
Add comment for dataSyncService (#8157) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-17 21:55:53 +08:00
bigsheeper 9601cb2ee6
Add failed streming unittest for query node (#7960) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-17 11:37:49 +08:00
bigsheeper c9b6a92366
Add release failed case unittest for query node (#7971) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-17 11:35:49 +08:00
bigsheeper b465e8f9a4
Remove redundant check and add addQueryChannel unittest for query node (#7944) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-16 09:47:48 +08:00
bigsheeper 18431c1549
Add historical failed unittest for querynode (#7941) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-15 15:25:49 +08:00
Cai Yudong 14ef405a6b
Rename chunk_size to chunk_rows for better code readability (#7953) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-15 15:15:52 +08:00
Cai Yudong a0fd2707cb
Support config SIMD type (#7942) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-15 12:57:48 +08:00
Cai Yudong e6c384b4d4
Move segcore chunk_size configuration to querynode.yaml (#7913) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-15 10:35:52 +08:00
bigsheeper 0b849d3089
Add insert node failed unittests for query node (#7914) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-15 10:11:48 +08:00
bigsheeper bc1f39bb55
Remove deprecated replica interface in query node (#7908) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-15 10:09:53 +08:00
bigsheeper dfc2535aaa
Use different schema generation functions (#7884) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-15 10:07:56 +08:00
bigsheeper 691e4c5bc9
Use definitional type instead of raw type (#7797) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-14 10:25:26 +08:00
bigsheeper 27beb033de
Add task unittest for query node (#7820) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-14 10:24:48 +08:00
bigsheeper 664d4ca923
Fix index loader test (#7805) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-14 10:21:08 +08:00
bigsheeper 73ab06c628
Decrease unittests execution time in query node (#7791) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-13 16:37:01 +08:00
bigsheeper ab7ded2202
Add loader and streaming unittests for query node (#7762) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-11 17:56:01 +08:00
bigsheeper 44bc0af063
Add node and queryCollection unittests for query node (#7719) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-11 17:26:01 +08:00
bigsheeper 3c01839d0d
Add impl and plan unittests for query node (#7712) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-11 17:22:01 +08:00
Cai Yudong 27dcf698d3
Support set segcore chunk_size via config file (#7635) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-11 14:40:01 +08:00
bigsheeper 0e81c62031
Check collection and plan before calling cgo functions (#7733) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-10 18:12:01 +08:00
bigsheeper cb5cd1cf12
Add index unittests for query node (#7674) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-10 17:11:09 +08:00
sunby db01040b53
Fix query request lost bug (#7715) 
Issue: #7633
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-10 16:34:44 +08:00
Cai Yudong 4e23ed9a18
Revert apply physical channels from pool (d5f7e358 and 3b960969) (#7687) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-10 13:00:00 +08:00
bigsheeper 8216478b3d
Add replica and insertNode unittests for query node (#7638) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-10 09:33:43 +08:00
bigsheeper 1a5e5989c8
Add segment unittests for query node (#7632) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-09 17:36:02 +08:00
bigsheeper 5906551f41
Add task unittest for query node (#7599) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-09 10:10:00 +08:00
bigsheeper d3a18a66b5
Add tSafeReplica and task scheduler unittests (#7587) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-08 20:50:00 +08:00
bigsheeper 50c677f72b
Remove useless message and add service time node unittest (#7565) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-08 17:10:15 +08:00
bigsheeper bcccd767ac
Add unittests for querynode (#7510) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-07 15:45:59 +08:00
bigsheeper b7103357fd
Load failed if collection beyonds the memory limit (#7434) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-07 11:35:18 +08:00
dragondriver 960741ee16
Support more detailed metrics information (#7472) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 19:35:47 +08:00
yukun 94272bba87
Support query by expression (#7386) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-09-03 17:12:55 +08:00
Cai Yudong 3b9609692b
Apply for msgstream from pool when creating collection (#7377) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 18:35:58 +08:00
Cai Yudong f4b53df727
Revert "Apply for msgstream from pool when creating collection (#7323)" (#7376) 
This reverts commit e6de86a433.
 2021-08-31 11:07:58 +08:00
Cai Yudong e6de86a433
Apply for msgstream from pool when creating collection (#7323) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 09:47:58 +08:00
godchen c333af0dcf
Add query node segment filter (#7303) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-28 10:12:00 +08:00
godchen 2dc2cb1a28
Fix collection nil when add query collection (#7311) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-27 17:51:56 +08:00
bigsheeper 14391fe9a2
Release flowgraphs properly (#7266) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-27 17:03:56 +08:00
bigsheeper 1e60cccaf9
Set inMemoryPercentage to 0 when releasing partition (#7310) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-27 16:51:11 +08:00
Xiangyu Wang c62f23e201
[skip ci]Remove neza2017 from reviewers (#7324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-27 00:27:36 +08:00
bigsheeper 653ec6cedb
Remove service time check in streaming search (#7253) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-24 15:07:52 +08:00
Cai Yudong 0ab70271a6
correct empty search result handle (#7244) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-24 11:13:52 +08:00
Xiaofan 58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-24 09:45:51 +08:00
godchen 8cc24e09c9
Make users get vectors without using local storage (#7134) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-18 16:30:11 +08:00
dragondriver e45cb6c644
Expose system topology information by proxy (#7137) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-08-18 10:12:10 +08:00
dragondriver 7eb8a665b2
Expose metrics of QueryCoord and QueryNodes (#7113) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-08-17 10:06:11 +08:00
xige-16 b035ec5c7b
Enable search when querynode only load segment (#7104) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-08-16 12:24:09 +08:00
Cai Yudong e771bda92f
optimize retrieve output vector code structure (#7102) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-14 11:18:10 +08:00
XuanYang-cn 6c5be83ea1
Refactor NewEtcdKV API (#7075) 
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 11:04:09 +08:00
Xiangyu Wang 5e78fd6a97
[skip ci]Update approvers (#7063) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-12 19:36:09 +08:00
Cai Yudong 6c75301c70
optimize search reduce logic (#7066) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-12 18:00:11 +08:00
congqixia 3f21e3d062
Fix querynode tt producer leak (#7059) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-12 14:16:08 +08:00
bigsheeper b9d3897deb
Return error when any failed detected in loadSegment (#7005) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-12 14:14:08 +08:00
godchen 7fc7623b92
Add vector data memroy existence check (#7044) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-11 19:50:09 +08:00
bigsheeper 4123deef9e
Return error message in grpc.LoadSegment in query node (#6908) 
* return load error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* init streaming meta in loadSegment

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix excluded init

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* improve impl

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-03 22:01:27 +08:00
Xiaofan aba234fa3f
Fix Msgstream exponential retry (#6807) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-03 10:39:24 +08:00
Cai Yudong fe50f97726
Use id to tell search result validation in proxy reduce stage (#6905) 
* use id to tell search result validation

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable test_search_binary_hamming_flat_index

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix merge retrieve result issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-02 10:25:49 +08:00
bigsheeper e8ea8b51d5
improve query node log (#6897) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-31 10:47:22 +08:00
bigsheeper eee0f1e077
Add mutex for segment of segCore in querynode (#6870) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-29 16:03:22 +08:00
godchen db94d7771f
Read vector from disk (#6707) 
* Read vector from disk

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix git action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix test error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix caculate error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change var name

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix len error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused code

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change bytes to float method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change float to bytes method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-07-24 09:25:22 +08:00
bigsheeper 21bc5810c0
remove redundant log of query node (#6768) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-23 20:09:33 +08:00
bigsheeper 2f44fc40a3
fix historical unittest (#6705) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-23 19:55:07 +08:00
Cai Yudong cbb4052fdb
Support search and query output fields using wildcard (#6671) 
* update wildcard polocy for search and query

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* mark xfail cases

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* mark xfail cases

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize debug log

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-21 14:52:12 +08:00
FluorineDog 234954931f
remove deprecated (#6623) 
* remove deprecated

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* fix timeout

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-20 15:18:08 +08:00
Cai Yudong a7b27db63e
update fillRetrieveResults parameter (#6598) 
* update fillRetrieveResults parameter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize fillVectorFieldData process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-17 15:17:30 +08:00
Cai Yudong a992dcf6a8
Support query return vector output field (#6570) 
* improve code readibility

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add offset in RetrieveResults

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add VectorFieldInfo into Segment struct

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add new interface for query vector

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update load vector field logic

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update load vector field logic

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fill in field name in query result

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add FieldId into FieldData

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add fillVectorOutputFieldsIfNeeded

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update data_codec_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add DeserializeFieldData

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* realize query return vector output field

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* disable query vector case

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-16 17:19:55 +08:00
Cai Yudong 1c29b42434
change output_fields to out_fields_id for RetrieveRequest (#6506) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-14 13:11:54 +08:00
bigsheeper e50d39dfe1
Query result get global sealed segments from ETCD (#6377) 
* get global sealed segments from ETCD

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix race error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-14 11:15:54 +08:00
Cai Yudong 724f10b9a0
Unify the usage of query and search (#6467) 
Unify the usage of query and search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-13 22:20:33 +08:00
xige-16 2b720fd2f7
Fix load hang when restart querynode many time in short time (#6351) 
* fix load hang when restart querynode many time in short time

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add multi queryNode ut

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add ut for restart querynode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* set queryCoord contex to load collection

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-07-13 14:16:00 +08:00
dragondriver bd317a5461
Check if field data is empty (#6404) 
* Check if field data is empty

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* FixBug: fast fail when output_fields contain fields that are not in schema

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-07-11 19:39:52 +08:00
neza2017 8b89744c6b
search with exp (#6409) 
* process float vector and binary vector field
on the search result

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add more debug info on etcd

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add debug info when grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-07-10 10:21:52 +08:00
Cai Yudong 21a9bcd844
Create default partition when receive CreateCollectionMsg (#6305) 
* create default partition when create collection

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* adjust sleep time for unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-06 09:16:03 +08:00
FluorineDog bf8b2be4a7
Deprecate num_groups to simplify search API (#6230) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-01 10:32:15 +08:00
zhenshan.cao ff4c6c7ff4
Fix travel timestamp and guarantee timestamp (#6234) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-30 21:02:13 +08:00
Cai Yudong a7a9eeeffe
Add time record for build index and search process (#6231) 
* add timerecord for build index process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add timerecord for query process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-30 19:46:14 +08:00
xige-16 ff93d1611f
merge retrive and search code in query node (#6227) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-30 17:50:15 +08:00
xige-16 f146d3825f
Fix search hang after querynode restart (#6212) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-30 17:48:19 +08:00
Cai Yudong 95135c1681
fix blob file sort error (#6184) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-29 13:24:15 +08:00
xige-16 6b65529086
Fix search error after releasePartition (#6135) 
* fix search error after releasePartition

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-26 16:08:11 +08:00
xige-16 8db95a63f6
Return error when search limit equal 0 or grater than 16384 (#6115) 
* return error when search limit equal 0 or grater than 16384

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-26 11:44:11 +08:00
bigsheeper 0d0c1901dd
Fix service time check of search in streaming (#6068) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-25 20:18:09 +08:00
yukun 39614aa8eb
Add rocksmq_path in config and paramtable (#6099) 
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 19:44:11 +08:00
yukun eb818d05ab
Remove vector fields from output fields (#6112) 
* Remove vector fields from output fields

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add empty_collection error_code

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 16:52:11 +08:00
bigsheeper e87372761a
Remove tSafe in release partition task and return max timestamp (#6078) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-24 19:06:34 +08:00
congqixia 10e5fc3382
Fix query node crash when retrieved 0 result (#6087) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-24 19:04:34 +08:00
yukun c8bf9078ae
Change pchannels to vchannel in retrieveResults (#6054) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-24 13:40:03 +08:00
yukun b72e4c6372
Merge search_collection and retrieve_colletion into query_collection (#6037) 
* Merge search_collection and retrieve_colletion into query_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix static-check

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-23 20:26:10 +08:00
bigsheeper dca24f313a
fix search timeout error (#6027) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-23 18:04:22 +08:00
bigsheeper 0f89f40488
Use virtual channel in search result and retrieve result (#6030) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-23 18:00:37 +08:00
xige-16 5417fee45f
fix releasePartition not work after loadCollection (#6025) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-23 17:44:12 +08:00
bigsheeper 47d89abaa0
get channels by static converting (#6015) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-23 17:26:09 +08:00
Cai Yudong a09a3a1905
rename service to coord (#6020) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-23 16:14:08 +08:00
godchen fd07a44dba
Fix error in retry refactor (#6000) 
* Fix error in retry refactor

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 11:48:06 +08:00
godchen 99be4c09a5
change retry (#5996) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 09:24:10 +08:00
bigsheeper b31bae351d
fix empty search result (#5979) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-22 16:52:08 +08:00
xige-16 2af1c27811
Rename queryservice to querycoordinator (#5931) 
* Rename queryservice to querycoordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix param table

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix unittest

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* service to coordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* set querycoord in ci

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-22 16:44:09 +08:00
xige-16 7039fb7c82
Fix the problem of stuck after loadBalance and loadFieldData error (#5960) 
* delete nodeInfo after nodeDown

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance can's stop

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load field data error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* contiue task loop after error in queryService

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-22 14:10:09 +08:00
yukun b2eb596c23
Change search result to columnBased and return target_entry (#5896) 
* Change search result to columnBased and return target_entry

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: fluorinedog <fluorinedog@gmail.com>

* synchronize the latest pymilvus-orm version

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change pymilvus-orm version to 40

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-21 20:18:13 +08:00
bigsheeper 2e615907ca
fix test_search_after_index_different_metric_type (#5921) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-21 18:46:15 +08:00
sunby 7f2d7c1da3
Rename dataservice to datacoordinator (#5912) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 18:22:13 +08:00
Cai Yudong 92e429d812
Rename IndexService to IndexCoord (#5932) 
* rename package indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix queryservice static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move distributed/indexservice to distributed/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move internal/indexservice to internal/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename MasterComponent to RootCoordComponent

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for queryservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for dataservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for datanode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for proxynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for querynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix rebase issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-21 17:28:03 +08:00
zhenshan.cao 958280ebb8
Add support for time travel (#5897) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-21 10:42:10 +08:00
bigsheeper bbd8a7e13a
Add released partitions and fix search error in empty partition (#5893) 
* rename flowgraphType to loadType

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* add load type and released partitions

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* filter released partition

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-19 18:38:07 +08:00
FluorineDog 255e3959af
support time travel (#5894) 
* support time travel

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-19 17:38:11 +08:00
XuanYang-cn ab7f642740
Add alias in paramtable (#5878) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-19 12:38:06 +08:00
xige-16 e4c51aae36
Add recovery logic for querynode and queryservice (#5843) 
* merge milvus/recovery2

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add recovery logic in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug smoke case

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add etcd to querynode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix release partition error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug querynode down and recovery

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add log

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix showCollection

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* skip smoke test search without insert

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-19 11:45:09 +08:00
sunby 33f5225968
Use physical channel to init msg stream (#5815) 
* Use physical channel to init msg stream

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix search collection error check (#1)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: bigsheeper <954206947@qq.com>
 2021-06-18 16:02:05 +08:00
bigsheeper 46151d203f
Fix tSafe cannot update correctly (#5820) 
* Fix tSafe cannot update correctly

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* skip test_load_partitions_release_collection

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* skip test_release_collection_during_searching

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-17 16:56:04 +08:00
yukun e9a8d1c404
Add vChannels in proxy for query results (#5802) 
* Fix proxynode for new retrieve logic

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove querynodenum from proxynode and querynode

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove QueryNodeIDList from proxy

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-16 20:15:59 +08:00
bigsheeper d6f7fbf19f
release collection synchronously (#5798) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-16 15:31:57 +08:00
bigsheeper bf39644457
Fix channels in query node and fix search (#5795) 
* fix add channel error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix tSafe vChannel key

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-16 12:05:56 +08:00
xige-16 97049ce293
use dataservice's getRecoveryInfo to get load and watch info (#5790) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-16 11:09:56 +08:00
yukun 5c18138a6b
Fix retrieve_collection return nil vchannel (#5787) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-15 20:07:30 +08:00
bigsheeper 58bf92a407
Add vChannel,pChannel in query node and get vChannel from master in query service (#5784) 
* add vChannel,pChannel in query node and get vChannel from master in query service

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-15 20:06:10 +08:00
yukun fceba63a33 Add segment and channel msgs in retrieve results (#5750) 
* Fix query hang bug (#5709)

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add segment and channel msgs in retrieve results

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-15 16:38:59 +08:00
zhenshan.cao 5c69c6902d Refactor proxy channelTimeTick and fix search timeout (#5760) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
zhenshan.cao 33a43f48c8 [skip ci] Fix bug: Search timeout and refactor client init logic (#5735) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:24:48 +08:00
xige-16 f2e551e9a0 Add asConsumer before seek (#5706) 
* add asConsumer before seek

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* get dmChannel from master

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* use pChannel instead in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-15 16:14:27 +08:00
FluorineDog bec9f2c182
Split segcore and plan proto for future feature (#5767) 
* Split segcore and plan proto for future feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-15 14:43:57 +08:00
bigsheeper cdbc6d2c94
Refactor query node and query service (#5751) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 12:41:40 +08:00
yukun e97936ba1c
Fix query hang bug (#5709) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-10 14:33:49 +08:00
bigsheeper 8aae0f7cc9
Refactor flow graph and load/watchDML in query node (#5682) 
* Refactor flow graph and load/watchDML in query node

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-09 11:37:55 +08:00
zhenshan.cao bc46e4780d
Fixbug:index is not loaded in some cases (#5633) 
* Fixbug:index is not created in some cases

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Delete unecessary print

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* fix create index

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-06 09:41:35 +08:00
zhenshan.cao 7dac20c35c
Add log for poxynode transform data and querynode segment insert (#5627) 
* Add log for poxynode transform data and querynode segment insert

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Decrease print times

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-05 16:21:36 +08:00
zhenshan.cao 03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
FluorineDog 9a90313390
Support GetEntityByIDs in CGo, fix segcore bugs (#5563) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-04 10:38:34 +08:00
Xiangyu Wang e80f06c030
Add retrieve process log (#5561) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:56:33 +08:00
Xiangyu Wang f791ccb246 Update OWNERS file 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:01:24 +08:00
yukun 57ece4a771
Fix retrieve_collection bugs (#5575) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-03 19:41:33 +08:00
zhenshan.cao b0b8f58192
Fix bug: search timeout (#5557) 
* Fix bug: search timeout

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Add log and fix unittest bug

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-03 14:58:34 +08:00
yukun dcb4161c9f
Fix retrieve_service (#5531) 
* Fix retrieve_service

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix retrieve_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-02 19:18:33 +08:00
yukun 48f35ffa97
Add publishRetrieveResult (#5501) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-31 18:08:32 +08:00
Xiangyu Wang ca0993c274 Update OWNERS 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-30 18:58:47 +08:00
Cai Yudong b414800d49
Update Seek interface (#5492) 
* update Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Seek for mqTtMsgStream

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use Retry in Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-29 23:21:34 +08:00
yukun cd549f812d
Add retrieve_service (#5481) 
* Add retrieve_service

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix for tSafeReplica

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-29 11:16:29 +08:00
bigsheeper 03f90ff28c
refactor collection replica and add tsafe replica in querynode (#5475) 
* refactor collection replica and add tsafe replica in querynode

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* static check

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* static check2

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* code format

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-05-28 15:40:32 +08:00
xige-16 fce792b8bf
Add historical and streaming module in querynode (#5469) 
* add historical and streaming

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix GetSegmentInfo

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* pass regression test

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2021-05-28 10:26:30 +08:00
neza2017 f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
neza2017 c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00
zhenshan.cao 3e73775cc1
Add ComputeProduceChannelIndexes for msgstream (#5364) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-24 08:30:09 +00:00
neza2017 e326eaad1e
master client,reconnect if grpc failed (#5360) 
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-24 01:44:49 +00:00
godchen 0eee7e5252
Change session (#5329) 
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-21 11:28:52 +00:00
sunby c1efa53779
Remove redundant fields in SegmentInfo and related structure (#5280) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-21 06:51:25 +00:00
Xiangyu Wang 23c4de0eb8
Flush statistics for all int64 fields (#5318) 
Resolves: #5262

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-20 10:38:45 +00:00
godchen e224d1e725
Add SessionManager (#5288) 
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-19 10:36:05 +00:00
godchen b74afd7a0e
Add service registration (#5189) 
Add service registration.
Part of Issue #5174.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-14 02:05:18 +00:00
dragondriver b7977698a8
Change the codec style of expr plan from text to binary (#5129) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 11:27:17 +00:00
dragondriver e5d4963ba0
Distinguish the dsl and expr in proxy and query node (#5118) 
Pass a parameter `IsExpr` from proxy to query node, so query 
node can create search plan according to the variable `IsExpr`.
It makes `segcore` unnecessary to distinguish `dsl` or `expr`
using dynamic exception. When `IsExpr` is set to true, query
node will call `CreatePlanByExpr` according to the passed
information about expression. Otherwise query node will keep
still to use `CreatePlan` according to the `dsl` information. At
the same time, this pr adds some unittests to `CreatePlanByExpr`,
these unittests translate already exist case with `dsl` to case with
`expr`.

Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 07:20:47 +00:00
FluorineDog 88f5642603
Add plan proto and support basic boolean expr parser (#5088) 
**What type of PR is this?**
- [x] Feature

**What this PR does / why we need it:**
This PR supports boolean expression as DSL.
1. The goal of this PR is to support predicates
    like `A > 3 && not B < 5 or C in [1, 2, 3]`. 
2. Defines `plan.proto`, as Intermediate Representation (IR) 
    used between go and cpp. 
3. Support expr parser, convert predicate expr to IR
    in proxynode, while doing static check there
4. Support IR to AST in cpp, enable the execution
 2021-04-29 08:48:06 +00:00
bigsheeper 88e1d651cb
Remove partition and segment ids properly in query node (#5090) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

**What type of PR is this?**

- [ ] API-change
- [x] BUG
- [ ] Improvement
- [ ] Documentation
- [ ] Feature
- [ ] Test and CI
- [ ] Code Refactoring

**Which issue(s) this PR fixes:**

Fixes #

**What this PR does / why we need it:**

Not Available
 2021-04-29 07:59:08 +00:00
bigsheeper 4197b094bf
Use internal dsl and placeholder group (#5033) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-25 06:08:28 +00:00
bigsheeper c4b379be8b
Set search graceful time from config (#5014) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-24 07:53:37 +00:00
bigsheeper 2b5bedf736
Use multiple search channels (#5018) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-24 02:44:28 +00:00
bigsheeper 00bb1014cd
Add search channels to query node (#5012) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-23 10:11:26 +00:00
bigsheeper dadb02db75
Remove unused params in query node (#4987) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-23 10:07:45 +08:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
sunby fad94fb576
Remove unsed configs (#4990) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-22 03:04:59 +00:00
xige-16 1165db75f6 Optimize search performance 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-19 19:30:36 +08:00
bigsheeper 71b8f309b2 Add licence to query node and query service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-19 13:47:10 +08:00
bigsheeper 08c5134286 Fix collection cannot be found when searching 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-16 17:37:50 +08:00
bigsheeper 9e7559b865 Make query node asynchronously load and release collection 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-16 14:40:33 +08:00
FluorineDog ce7a5ea699 Support segcoreinit 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-16 14:02:49 +08:00
XuanYang-cn e281f6a4a1 Remove useless configs in config files 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-13 10:04:39 +08:00
godchen deba964590 Change name of kvBase 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-12 18:09:28 +08:00
bigsheeper 39458697c7 Use asynchronous functions of load and release 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-12 09:18:43 +08:00
xige-16 22eda19bc4 Fix mem leak when load index 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-10 10:10:53 +08:00
sunby 751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
xige-16 332b14c915 Delete querynode’s redundant load IO 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-07 18:29:19 +08:00
bigsheeper fe3fa8dd1d Fix search service unittest 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-01 15:39:23 -05:00
Xiangyu Wang 87a1e0b662 Reorganize msgstream 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-02 13:48:25 +08:00
xige-16 c10b36f188 fix getting wrong offset when segment preinsert 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-31 16:16:58 +08:00