milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,553 Commits
130 Branches
115 Tags
562 MiB
Commit Graph

1341 Commits (b2f549d6a720858494b5d6d937c12beb6d520ee6)

Author SHA1 Message Date
XuanYang-cn 7a8b34ee9f Remove not using code in DataNode (#5476) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby a14c35274f Fix bugs in dataservice (#5457) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn 55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
dragondriver 59a3d83f97 Replace insertChannelsMap with channelsMgr (#5453) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia 607a8ffa1f Update WatchDmChannels signature (#5447) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby cfdd98b70f Fix recovery branch after rebasing master (#5444) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby 07c6a4a669 Add dataservice register and discovery (#5435) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia e3956ad13f Change SaveBinLogPaths field2Path to repeated (#8) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
yangxuan ef563fda8f Updata flushe procedure 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby c53afee616 DataNode scales flowgraph 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby b9cf4c1095 Fix recovery (#5417) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby 81ac20143b Add persistency of datanode cluster (#5387) 
We save channels registered in datanode in etcd and restore the cluster info after
restarting.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby 3662b3f0e5 Refactor dataservice (#5400) 
Pick up grpc calls to grpc_handler.go and refactor code format

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:01 +08:00
congqixia ef7339ceda DataService stores flush msgstream pos at VChannel granularity (#5397) 
SaveBinLogPaths saves stream positions at VChannel granularity

resolves #5396

Signed-off-by: Congqi Xia congqi.xia@zilliz.com
 2021-06-15 15:54:57 +08:00
FluorineDog bec9f2c182
Split segcore and plan proto for future feature (#5767) 
* Split segcore and plan proto for future feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-15 14:43:57 +08:00
bigsheeper cdbc6d2c94
Refactor query node and query service (#5751) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 12:41:40 +08:00
FluorineDog 4165b761c9
fix not in bug (#5741) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-11 17:08:25 +08:00
FluorineDog 006dae35c3
fix retrieve bug (#5727) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-11 00:06:55 +08:00
yukun e97936ba1c
Fix query hang bug (#5709) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-10 14:33:49 +08:00
bigsheeper 8aae0f7cc9
Refactor flow graph and load/watchDML in query node (#5682) 
* Refactor flow graph and load/watchDML in query node

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-09 11:37:55 +08:00
yukun 313658767d
Add field_name and type in retrieve result (#5665) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-08 16:26:36 +08:00
Xiangyu Wang 68f703f983
Add Query impl in proxynode (#5644) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-07 17:21:36 +08:00
sunby 50b3d39370
Add GetRecoveryInfo interface and proto (#5598) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-07 14:16:36 +08:00
zhenshan.cao bc46e4780d
Fixbug:index is not loaded in some cases (#5633) 
* Fixbug:index is not created in some cases

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Delete unecessary print

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* fix create index

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-06 09:41:35 +08:00
shengjun.li 8ead67ef95
IVF_SQ8 and IVF_PQ cannot be built on multiple GPUs (#5624) 
* IVF_SQ8 and IVF_PQ cannot be built on multiple GPUs

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* index created by std::make_shared

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-05 17:19:34 +08:00
zhenshan.cao 7dac20c35c
Add log for poxynode transform data and querynode segment insert (#5627) 
* Add log for poxynode transform data and querynode segment insert

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Decrease print times

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-05 16:21:36 +08:00
cai.zhang 5037646cbd
Check dsl whether is None (#5616) 
* Check dsl whether is None

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>

* Format code

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-05 14:57:34 +08:00
yukun 45e7d26b01
Fix retrieve when output_fields is nil (#5623) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-05 10:57:34 +08:00
FluorineDog 1ff4764ba6
Fix wrong primary key (#5604) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-04 18:35:36 +08:00
zhenshan.cao 03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
Cai Yudong 2ba93deb5e
Support master get segment info channel name from config file (#5606) 
* optimize msgstream unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* get data service segment info channel name from config file directly

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-04 16:28:34 +08:00
Xiangyu Wang afa5da7df0
Add query rpc into proto (#5592) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-04 12:02:34 +08:00
FluorineDog 9a90313390
Support GetEntityByIDs in CGo, fix segcore bugs (#5563) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-04 10:38:34 +08:00
Xiangyu Wang e80f06c030
Add retrieve process log (#5561) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:56:33 +08:00
Xiangyu Wang f791ccb246 Update OWNERS file 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:01:24 +08:00
yukun 57ece4a771
Fix retrieve_collection bugs (#5575) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-03 19:41:33 +08:00
dragondriver 99699b2021
Support to show collections which are loaded into memory (#5567) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-03 19:09:33 +08:00
godchen 0610ceb3dc
Fix ut error (#5573) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-03 18:44:33 +08:00
dragondriver e76830ab09
Support column-based insert request in proxy (#5560) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-03 15:03:34 +08:00
zhenshan.cao b0b8f58192
Fix bug: search timeout (#5557) 
* Fix bug: search timeout

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Add log and fix unittest bug

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-03 14:58:34 +08:00
Cai Yudong 4018dfc5b7
Support data segment channel seek to the break point when system start (#5540) 
* add DecodeDdOperation

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* reload MsgPosition from etcd and seek when system start

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename const string

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add msg into dataSegmentChannel to test seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix seek test

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back distributed/masterservice/masterservice_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-02 22:36:41 +08:00
yukun dcb4161c9f
Fix retrieve_service (#5531) 
* Fix retrieve_service

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix retrieve_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-02 19:18:33 +08:00
Cai Yudong d5cd561449
Add metrics for master (#5515) 
* metric

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* metric

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add MasterDDChannelTimeTick for metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add metrics for all master grpc

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* register metrics

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add metrics register

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add comments for codacy check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-01 11:04:31 +08:00
yukun 48f35ffa97
Add publishRetrieveResult (#5501) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-31 18:08:32 +08:00
godchen 918458a1be
Add msgstream seek (#5483) 
* Add msgstream seek

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change test name

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove file

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* resolve conversation

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-31 17:46:30 +08:00
Cai Yudong 5ac726b7b9
use NewTicker instead of NewTimer (#5503) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-31 14:44:30 +08:00
Cai Yudong c468481a36
Optimize MqTtMsgStream and unittest (#5498) 
* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add getTsMsgFromConsumerMsg

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename some variables for better readability

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add chanWaitGroup

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add chanTtMsgTime

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename lastTimeStamp to currTimeStamp

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add comments

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* support unsynced ttMsg

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add TestStream_PulsarTtMsgStream_3

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* remove debug log

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* change ttMsg logic back

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix ci block issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix codacy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-31 11:31:30 +08:00
zhenshan.cao 4af2cacb28
Refactor IndexService and IndexNode: remove redundant serviceID (#5491) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-31 10:32:30 +08:00
Xiangyu Wang ca0993c274 Update OWNERS 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-30 18:58:47 +08:00
Cai Yudong b414800d49
Update Seek interface (#5492) 
* update Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Seek for mqTtMsgStream

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use Retry in Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-29 23:21:34 +08:00
zhenshan.cao 67c4c915b7
Move debugging code of index query to a separate module (#5490) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-29 17:07:29 +08:00
Xiangyu Wang 95e762bb21
Add logs in retrieve request process (#5460) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-29 11:19:30 +08:00
yukun cd549f812d
Add retrieve_service (#5481) 
* Add retrieve_service

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix for tSafeReplica

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-29 11:16:29 +08:00
bigsheeper 03f90ff28c
refactor collection replica and add tsafe replica in querynode (#5475) 
* refactor collection replica and add tsafe replica in querynode

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* static check

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* static check2

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* code format

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-05-28 15:40:32 +08:00
FluorineDog b1a9aea6a6
support get entity by ids in segcore (#5456) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-05-28 10:39:30 +08:00
xige-16 fce792b8bf
Add historical and streaming module in querynode (#5469) 
* add historical and streaming

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix GetSegmentInfo

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* pass regression test

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2021-05-28 10:26:30 +08:00
yukun c63524deb9
Add dablooms (#5440) 
* Add dablooms cwrapper

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix undefined reference bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add cwrapper_dablooms_build shell script

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-28 10:23:30 +08:00
cai.zhang a606ab9c67
Add power-off restart logic for IndexService and IndexNode (#5395) 
Add power-off restart logic for IndexService and IndexNode


Signed-off-by: <xiaocai2333> <cai.zhang@zilliz.com>
 2021-05-27 14:24:29 +00:00
godchen 6e3a0d2a8e
Add queryservice grpc retry (#5437) 
Add queryservice grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-27 10:44:36 +00:00
zhenshan.cao 5ac8b8acc6
Add comments to proto (#5452) 
Specify which fields must be set by SDK

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-27 08:34:31 +00:00
godchen 220c12e12f
Add indexservice grpc retry (#5425) 
Add indexservice grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-27 07:12:13 +00:00
godchen 279d77020a
Change proxy grpc retry (#5431) 
Change proxy grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-27 07:10:39 +00:00
xige-16 3630eec92c
Fix receving wrong msg after seek (#5441) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-05-27 02:38:37 +00:00
godchen 9aa1fd25fa
Add indexnode grpc retry (#5418) 
Add indexnode grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-27 02:30:11 +00:00
neza2017 f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
godchen 5fd92ec80c
Add querynode grpc retry (#5428) 
Add querynode grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-26 11:37:34 +00:00
godchen bfa683ce79
Change dataservice grpc retry (#5411) 
Change dataservice grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-26 10:55:05 +00:00
godchen 5f0f840782
Remove datanode client code (#5427) 
Remove datanode client code.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-26 10:41:37 +00:00
yukun a3fb1356de
Fix rocksmq seek error (#5432) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>

In milvus-standalone, when seek is called, rocksmq won't put a
mutex in MsgMutex channel, which cause that the consume
process hang. So the seek function in rocksmq needs to 
notify the consumer to consume.
 2021-05-26 09:31:09 +00:00
Xiangyu Wang e021f5e670
Support to use json to construct dummy request (#5430) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-26 08:57:32 +00:00
godchen 200801271d
Fix GetSession error (#5401) 
Fix GetSession error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-25 12:15:46 +00:00
zhenshan.cao 6766169878
Refactor repack logic for insertion (#5399) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-25 11:53:15 +00:00
godchen 17cabfbed3
Add datanode client retry (#5394) 
Add datanode client retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-25 07:47:08 +00:00
neza2017 c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00
Xiangyu Wang 88b42304ef
Add dummy RPC (#5376) 
Adding a dummy RPC makes it easier to develop new RPCs in the future.
When developing a new function, the SDK is often implemented last. 
At this time, this is helpful for debugging.

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-25 06:44:43 +00:00
Cai Yudong dab536a6cf
Rename some field names in master for better code readability (#5389) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-25 06:03:06 +00:00
Cai Yudong 4058350e30
update AddIndex not return error when fail (#5372) 
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-25 03:42:23 +00:00
yukun 49443e8a33
Add retrieve taskscheduler implementation (#5353) 
See also: #5257 

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-24 10:19:43 +00:00
godchen 042a061123
Fix session util error (#5370) 
Fix session util error.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-24 08:31:06 +00:00
zhenshan.cao 3e73775cc1
Add ComputeProduceChannelIndexes for msgstream (#5364) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-24 08:30:09 +00:00
shengjun.li b688cc5f51
Improve IVF query on GPU when no entity deleted (#5359) 
Not to call getListIndex() if bitset is empty.

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-05-24 07:38:56 +00:00
Cai Yudong 39f8c9dd56
Save msg start position and end position when AddSegment and AddIndex (#5358) 
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-24 06:19:52 +00:00
FluorineDog 3f42a9ed12
reduce unittest time (#5363) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-05-24 03:54:06 +00:00
godchen dd736ee8ba
Change session doc (#5355) 
Change session doc

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-24 03:53:07 +00:00
zhenshan.cao 4e1b12269b
Fix bug: rand.Seed reset and generate duplicate number (#5361) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-24 02:50:37 +00:00
dragondriver 457e964e41
Add channelsTimeTicker to proxy (#5339) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-24 01:49:28 +00:00
neza2017 e326eaad1e
master client,reconnect if grpc failed (#5360) 
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-24 01:44:49 +00:00
godchen d189bf4d33
Change pulsar version (#5352) 
Change pulsar version.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-22 03:36:49 +00:00
zwd1208 7d49096e8c
Add prometheus metrics (#5342) 
Add prometheus metrics for components

Resolves: #5325

Signed-off-by: Weida Zhu <weida.zhu@zilliz.com>
 2021-05-21 13:48:04 +00:00
congqixia 4aa6a8fe18
Dataservice use lastest grpc design and saves ddl/dml pos pair (#5348) 
Dataservice use lastest grpc design and saves ddl/dml pos pair

* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)

See also #5220 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-21 11:38:58 +00:00
godchen 0eee7e5252
Change session (#5329) 
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-21 11:28:52 +00:00
sunby 02f2348500
Save segment allocation status to meta (#5184) 
To ensure segments being flushed correctly, we need to load 
the segment allocation status from meta after starting the service.
 2021-05-21 10:30:41 +00:00
congqixia 7a82462da8
DataService publishes flush-completed message into its channel (#5320) 
DataService publishes flush-completed message  into related msgstream

See also: #5220 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-21 08:54:29 +00:00
yukun 156d5b9f73
Add RetrieveTask PostExecute function (#5336) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>

See also: #5257
 2021-05-21 08:39:28 +00:00
congqixia 46790d9c68
Fix issue 5333 uninitialized return value (#5338) 
Fix uninitialized named value `meta` caused crash

Resolves #5333 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-21 08:21:08 +00:00
Cai Yudong 761350e44d
Refactor timetick msg handling between proxy node and master (#5321) 
Proxy node send channel timetick msg to master, master reduce timetick
msg for each channel and send the min timetick to msg stream.

Resolves: #5276 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-21 08:08:12 +00:00
sunby c1efa53779
Remove redundant fields in SegmentInfo and related structure (#5280) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-21 06:51:25 +00:00
zhenshan.cao 70241a8bf9
Cancel local timestamp caching logic (#5327) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-21 05:11:21 +00:00
dragondriver f6ab3c81df
Add channelsMgr to proxy (#5323) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-21 01:52:20 +00:00
Xiangyu Wang 23c4de0eb8
Flush statistics for all int64 fields (#5318) 
Resolves: #5262

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-20 10:38:45 +00:00
godchen 913712fbea
Improve session manager (#5310) 
Improve session manager.
Issue #5174

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-20 07:07:25 +00:00
yukun f18dfb4ff6
Add RetrieveTask implementation (#5313) 
Resolves: #5257 

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-20 07:02:31 +00:00
neza2017 c0daf8e41d
set timestamp on dd requests (#5298) 
- set dd request's time tick on master
- send time tick into dd channel

See also: #5291

Signed-off-by: yefu.chen yefu.chen@zilliz.com
 2021-05-20 06:14:14 +00:00
congqixia 38f5b0826f
Add SaveBinlogPaths grpc in dataservice (#5277) 
* Add SaveBinlogPaths grpc in dataservice

* fix merged compile error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-20 11:34:45 +08:00
XuanYang-cn 1c779f3efd
Refactor ddl binlog flush (#5303) 
DDL won't auto-flush now

See also: #5289 , #5220 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-05-19 11:42:07 +00:00
yukun a7bb701f73
Add Retrieve method grpc definition (#5294) 
See also: #5253 

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-19 10:45:15 +00:00
godchen e224d1e725
Add SessionManager (#5288) 
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-19 10:36:05 +00:00
quicksilver 3db653f2d0
Update cpp thirdparty source URL (#5295) 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

Resolves: #5293
 2021-05-19 09:39:29 +00:00
congqixia d5ab8ac345
Restore flush and stats stream pos (#5284) 
Restore segment flush stream & statistic stream to last success pos

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-19 06:13:53 +00:00
XuanYang-cn fc630bc1c8
Correct auto-flush behaviour (#5282) 
Before this PR, DataNode considered auto-flush a valid flush
complete. It's wrong. So I open this PR to correct this behaviour
in DataNode.

Now binlog paths from auto-flush will be buffered in replica,
waiting until the manul flush to save into etcd all together.

See also: #5220, #5268
A follow up job of #5271 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-05-18 11:45:00 +00:00
godchen ccc0130bb4
Add exclusive parameter (#5278) 
Add exclusive parameter.
Issue #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-18 11:07:27 +00:00
neza2017 d68ee3fbc2
query base on timeline (#5279) 
query base on timeline, add timestamp on these queries:
- ShowCollections
- HasCollection
- DescribeCollection

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-18 09:12:17 +00:00
XuanYang-cn c3ac1375d8
Add buffer function for auto flush (#5271) 
Auto-flush of a segment is not considered flush-completed.
So we need to buffer binlog paths generated by auto-flush.

See also: #5268 #5220 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-05-18 08:33:05 +00:00
neza2017 49f6542b1d
query master service's with timestamp (#5275) 
let meta support snapshot
so collection could query meta with timestamp

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-18 06:18:02 +00:00
godchen cca75583a6
Add session util (#5269) 
Add session util.
Issue #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-18 03:39:21 +00:00
Cai Yudong e8ceb37f81
Add ShardsNum when create collection (#5245) 
Updata PhysicalChannelNames and VirtualChannelNames based
on ShardsNum when create collection, and update collVChan2Chan
map.

Resolves: #5240 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-17 11:15:01 +00:00
Cai Yudong 001795ee70
Optimize CreateIndex routine in master service (#5238) 
Remove CreateIndexTask struct and indexTaskQueue

Resolves: #5228 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-15 10:08:08 +00:00
Cai Yudong 08bb1b2ec3
Resend unsuccessful ddMsg when master start (#5214) 
Resend unsuccessful ddMsg when master start

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-14 21:26:06 +08:00
XuanYang-cn 9151e13480
Remove not using dependence and proto (#5226) 
- FlushMsg proto message is no longer in use

- We don't depend on "github.com/prometheus/common/log"

Signed-off-by: yangxuan xuan.yang@zilliz.com
 2021-05-14 08:50:55 +00:00
yukun 4e6e502df1
Add data sorter in storage (#5200) 
GetEntityByID needs the flushed segment to be sorted by RowID field, 
then do binary search to get the target id and entities.

See also: #5177 

Signed-off-by: fishpenguin kun.yu@zilliz.com
 2021-05-14 02:59:49 +00:00
godchen b74afd7a0e
Add service registration (#5189) 
Add service registration.
Part of Issue #5174.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-14 02:05:18 +00:00
Cai Yudong 7b83b11e95
Save ddmsg info and type into ETCD (#5173) 
Save ddmsg info and type into ETCD to support re-send ddmsg 
when system restart or recover from failure (#5172)

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-12 07:33:53 +00:00
dragondriver 64ab4d740b
Add row_count to partition statistics (#5162) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-10 09:39:08 +00:00
sunby a00422bac2
Remove unused interfaces in dataservice (#5157) 
Remove unused interfaces in dataservice

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-10 09:03:24 +00:00
Cai Yudong 6f3a460b4e
Add pulsar seek example in pulsar_client_test.go (#5154) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-10 02:32:10 +00:00
dragondriver df71f4c77c
Migrate softTimeTickBarrier from proxyservice to timesync (#5146) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-10 01:28:59 +00:00
XuanYang-cn eb557b289b
Add docs in package datanode (#5117) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-05-08 07:24:12 +00:00
Cai Yudong 5f0006d0f9
Improve code coverage for distributed/datanode (#5133) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-08 06:07:04 +00:00
yukun ecaef24fea
Change should_not to must_not in dsl (#5131) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-07 11:52:24 +00:00
FluorineDog b1974aede2
Merge knowhere unittests (#5139) 
Re-enable knowhere unittests and merge them all together. 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-05-07 11:50:13 +00:00
dragondriver b7977698a8
Change the codec style of expr plan from text to binary (#5129) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 11:27:17 +00:00
Cai Yudong d66a5acca3
improve code coverage for distributed/dataservice (#5123) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
improve code coverage for distributed/dataservice
 2021-05-07 07:43:52 +00:00
dragondriver e5d4963ba0
Distinguish the dsl and expr in proxy and query node (#5118) 
Pass a parameter `IsExpr` from proxy to query node, so query 
node can create search plan according to the variable `IsExpr`.
It makes `segcore` unnecessary to distinguish `dsl` or `expr`
using dynamic exception. When `IsExpr` is set to true, query
node will call `CreatePlanByExpr` according to the passed
information about expression. Otherwise query node will keep
still to use `CreatePlan` according to the `dsl` information. At
the same time, this pr adds some unittests to `CreatePlanByExpr`,
these unittests translate already exist case with `dsl` to case with
`expr`.

Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 07:20:47 +00:00
xige-16 1e7205fa03
fix querynode's port not set correctly when conflict with other node (#5066) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-05-07 12:03:53 +08:00
FluorineDog 396b3f33e9
Support TermExpr, NotExpr, LogicalExpr (#5096) 
1. Support Term, like `A in [1, 2, 3]`
2. Support Not, like `! A < 3`
3. Support logical combination, like `A < 3 && B > 5 or C == 0`

Type: Feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-04-30 07:19:52 +00:00
shengjun.li b46ae44087
Zero size allocation in StackDeviceMemory (#5100) 
When the remaining space is empty, the zero size allocation will
get the pointer `end_`. However, `cudafree` the pointer `end_` will
cause to crash.

Fix: #5078 #4770 #4412 #4340 #3646

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-04-30 04:21:02 +00:00
FluorineDog 88f5642603
Add plan proto and support basic boolean expr parser (#5088) 
**What type of PR is this?**
- [x] Feature

**What this PR does / why we need it:**
This PR supports boolean expression as DSL.
1. The goal of this PR is to support predicates
    like `A > 3 && not B < 5 or C in [1, 2, 3]`. 
2. Defines `plan.proto`, as Intermediate Representation (IR) 
    used between go and cpp. 
3. Support expr parser, convert predicate expr to IR
    in proxynode, while doing static check there
4. Support IR to AST in cpp, enable the execution
 2021-04-29 08:48:06 +00:00
bigsheeper 88e1d651cb
Remove partition and segment ids properly in query node (#5090) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

**What type of PR is this?**

- [ ] API-change
- [x] BUG
- [ ] Improvement
- [ ] Documentation
- [ ] Feature
- [ ] Test and CI
- [ ] Code Refactoring

**Which issue(s) this PR fixes:**

Fixes #

**What this PR does / why we need it:**

Not Available
 2021-04-29 07:59:08 +00:00
XuanYang-cn cebdfb34f7
Enable multiple datanode (#5068) 
Now it's able to start multiple datanode, however, they will process the same insert data.
Further changes are needed to truly enable multiple datanode:

While registering, dataservice need to return different insert channels for different datanode.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-04-28 18:43:48 +08:00
godchen 004598f796
Remove field in desc index interface. (#5080) 
* remove field in desc index interface

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* trigger GitHub actions

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-28 09:48:38 +00:00
godchen 94f0966225
Add get index build progress interface implementation (#5067) 
Add get index build progress interface implementation.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-28 03:15:28 +00:00
godchen 060f7ca0d2
Add get index build progress proto (#5049) 
Add get index build progress proto.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-27 07:46:45 +00:00
shengjun.li a3e4339027
improve rhnsw (#5059) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-04-27 06:17:52 +00:00
neza2017 262c484cec
the return struct of "DescribeIndex" should contain "filed_name" (#5055) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-27 02:30:55 +00:00
neza2017 d7c0a72611
index name is unique in collection scope (#5046) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-26 15:53:47 +08:00
Cai Yudong 74f457bd14
add more testcase to improve code coverage for data service (#5041) 
* fix TestChannel run error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* comment validateAllocRequest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add test APIs

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-26 01:45:54 +00:00
bigsheeper 4197b094bf
Use internal dsl and placeholder group (#5033) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-25 06:08:28 +00:00
sunby 4d03da9c4f
Remove `SetSegmentState` and seal segment in segment allocator (#4994) 
Segment should be sealed once it exceeds the limited size. This operation
should be done in segment allocator. Watcher only check the status to
decide whether the segment will be flushed.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-25 01:51:57 +00:00
neza2017 ada58b5bde
add unit test for master's grpc server, the coverage of package "internal/distributed/masterservice" increase from 46.6% to 82.9% (#5019) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-24 09:23:35 +00:00
bigsheeper c4b379be8b
Set search graceful time from config (#5014) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-24 07:53:37 +00:00
sunby b855c283cb
Remove meaningless log test (#5025) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-24 05:07:58 +00:00
Cai Yudong 5a75e23795
improve data service code coverage (#5020) 
* optimize TestAssignSegmentID

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update mockMasterService to improve code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename client to kvClient in Server

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update server.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-24 03:29:15 +00:00
bigsheeper 2b5bedf736
Use multiple search channels (#5018) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-24 02:44:28 +00:00
bigsheeper 00bb1014cd
Add search channels to query node (#5012) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-23 10:11:26 +00:00
shengjun.li a2875f9d95
Update knowhere (#5006) 
Import performance of ivf::train and hnsw, and fix bugs
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-04-23 18:03:49 +08:00
Cai Yudong 111376ba40
improve code coverage for data service (#4998) 
* rename mock.go to mock_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add allocator_test.go to improve code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update meta_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update watcher_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-23 02:15:30 +00:00
godchen 4f7e393bfd
Remove useless config (#4997) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-23 10:10:46 +08:00
bigsheeper dadb02db75
Remove unused params in query node (#4987) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-23 10:07:45 +08:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
sunby fad94fb576
Remove unsed configs (#4990) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-22 03:04:59 +00:00
xige-16 3bb69430cb
delete contex timeout in queryService's load (#4986) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-22 09:23:49 +08:00
godchen 139b334732
Add kv etcd unittest and remove useless code (#4974) 
* Add kv etcd unittest and remove useless code

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-21 18:45:35 +08:00
Cai Yudong 51cae2c9ba
add datanode unittest to improve code coverage (#4982) 
add datanode unittest to improve code coverage 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-21 18:41:37 +08:00
xige-16 1165db75f6 Optimize search performance 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-19 19:30:36 +08:00
sunby b79a408491 Remove unused config item in dataservice and datanode 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-19 19:28:45 +08:00
cai.zhang 9ccaff8005 Add copyright 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-04-19 19:28:11 +08:00
FluorineDog 43b39cc9c9 Fix memory leak, temp solution 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-19 16:39:23 +08:00
XuanYang-cn 6deba1faa5 Add license in datanode package 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-19 15:16:33 +08:00
zhenshan.cao b1f664d04f Add copyright 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-04-19 15:15:33 +08:00
cai.zhang 8a037a404e Add Copyright for indexnode and indexservice 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-04-19 13:50:12 +08:00
bigsheeper 71b8f309b2 Add licence to query node and query service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-19 13:47:10 +08:00
Xiangyu Wang c731e2df78 Add license header in util 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-19 13:42:47 +08:00
sunby 0f2f46f39d Add copyright 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-19 11:35:38 +08:00
godchen 0dfcb90881 Add storage copyright 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-19 11:32:24 +08:00
xige-16 51b2b45e03 Add copyright for msgStream 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-19 11:30:19 +08:00
FluorineDog 6059558698 Add license files 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-19 11:16:16 +08:00
yukun 89c4bf82a0 Add copyright in kv module 
Signed-off-by: kun yu <kun.yu@zilliz.com>
 2021-04-19 11:13:52 +08:00
neza2017 83647e8503 Add copyright in master service module 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-19 11:12:56 +08:00
godchen a5ad70a5ab Add unittest for storage 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-19 10:36:19 +08:00
Cai Yudong a2041abab4 improve code coverage for datanode 
improve code coverage for collection_replica_test.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-19 10:34:56 +08:00
dragondriver dc7db73251 Add copyright annotations to proxynode and proxyservice 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-04-19 10:09:43 +08:00
bigsheeper 08c5134286 Fix collection cannot be found when searching 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-16 17:37:50 +08:00
sunby 94c3fc9321 Add unit tests in dataservice 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-16 16:30:55 +08:00
cai.zhang a54097708d Adjust the DropIndex logic of the indexservice 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-04-16 15:37:13 +08:00
bigsheeper 9e7559b865 Make query node asynchronously load and release collection 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-16 14:40:33 +08:00
FluorineDog ce7a5ea699 Support segcoreinit 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-16 14:02:49 +08:00
xige-16 78155d3959 Refactor load Release to async call in query service 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-15 15:15:46 +08:00
XuanYang-cn e281f6a4a1 Remove useless configs in config files 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-13 10:04:39 +08:00
sunby a7dac818ee Refactor datanode register 
Dataservice should not be stalled during registering datanode.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-13 09:47:02 +08:00
godchen deba964590 Change name of kvBase 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-12 18:09:28 +08:00
sunby 61edc53faf Refactor data service segment allocator 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-12 16:35:51 +08:00
neza2017 e760605187 Add unit test for master service, the coverage of meta table reachs 100% 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-12 15:03:23 +08:00
Cai Yudong 687d48b75e Improve code readibility 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-04-12 14:11:06 +08:00
godchen 8204546bcd Add unit test for binlog module 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-12 14:07:46 +08:00
bigsheeper 39458697c7 Use asynchronous functions of load and release 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-12 09:18:43 +08:00
neza2017 15f26f653e Add unit test for master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-10 10:53:58 +08:00
xige-16 22eda19bc4 Fix mem leak when load index 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-10 10:10:53 +08:00
neza2017 56d367b1e9 Add unit test for master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-09 16:10:12 +08:00
Cai Yudong 952f70af2a Clean makefile 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-04-09 14:07:03 +08:00
sunby 751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
yukun 964e7e0a28 Add unittest for mqclient 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-04-08 20:05:33 +08:00
neza2017 8dd3e1e28f Add unit test for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-08 17:31:39 +08:00
dragondriver 57831b9978 Add unittest to insertChannelsMap in proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-04-08 15:41:28 +08:00
neza2017 a250eb370f Check request type and add unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-08 15:26:18 +08:00
sunby 6e70ce3f66 Add cluster unit tests 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-08 15:08:34 +08:00
sunby 1d7195e036 Add minio and etcd stats watcher 
Our system needs to watch the performance of etcd and minio. For now I just add the creation stats.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-08 14:14:43 +08:00
XuanYang-cn 46d27e37d7 Skip install gtest when build with test 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-08 10:01:18 +08:00
dragondriver 846fd8a154 Add unittest to conf adapter in proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-04-08 09:51:04 +08:00
xige-16 332b14c915 Delete querynode’s redundant load IO 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-07 18:29:19 +08:00
FluorineDog 1446cd5453 Fix flat unsupported bug 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-07 10:39:35 +08:00
dragondriver bfb7ca0df6 Add unittest to time tick logic in proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-04-07 10:06:17 +08:00
dragondriver decc80a525 Add unittest to proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-04-06 14:12:57 +08:00
bigsheeper fe3fa8dd1d Fix search service unittest 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-01 15:39:23 -05:00
Xiangyu Wang 87a1e0b662 Reorganize msgstream 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-02 13:48:25 +08:00
Xiangyu Wang 8b62790017 Reorganize mqclient 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-02 10:01:11 +08:00
neza2017 d7f5c6b1ef Add benchmark for timtick 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-01 18:05:43 +08:00
yukun cdc96d1c32 Add unittest for msgstream_impl 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-04-01 17:29:21 +08:00
zhenshan.cao c6950eb7eb Add log for allocator 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-04-01 13:37:18 +08:00
neza2017 a532444da7 Add log when alloc time stamp failed 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-01 11:25:30 +08:00
dragondriver 6796c7bfe7 Add unittest to function util 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-31 18:22:17 +08:00
xige-16 c10b36f188 fix getting wrong offset when segment preinsert 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-31 16:16:58 +08:00
yukun 18e0581d5c Destroy readOptions after use 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-31 14:10:07 +08:00
zhenshan.cao 7bad0a01e9 Fix bug: unmarshal got unexpected EOF 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-31 13:55:51 +08:00
yukun 604f96ed2b Replace consumers map by sync.map 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-31 10:05:24 +08:00
xige-16 a6f1de036b Optimize search performance in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-30 22:16:58 +08:00
zhenshan.cao efbf4452c5 Add log for port changed or proto unmarshal error 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-30 20:19:30 +08:00
godchen 8f58d5641b Perfect trace and add trace init 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-30 16:36:34 +08:00
Xiangyu Wang 8d66388b8c Add logs for querynode 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-30 15:00:14 +08:00
xige-16 e823b56040 Add retry for seek function 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-30 13:53:49 +08:00
Xiangyu Wang 72917467ec Change singlenode logs output 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-30 10:52:42 +08:00
XuanYang-cn 91f5c48037 Fix datanode goroutine leak 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-30 09:47:27 +08:00
sunby f8e879b423 Fix high cpu usage in singlenode 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-29 17:48:15 +08:00
sunby ff8ebd5f42 Fix bug of estimating record size 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-29 15:14:51 +08:00
neza2017 71614313b0 Fix master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-29 09:50:52 +08:00
neza2017 536a0e33d6 Fix meta table, dead lock 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-27 19:46:00 +08:00
yukun c4f9a6f22b Fix get position wrong 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-27 17:39:12 +08:00
sunby a78cd3e2a7 Add proxy service timetick loop to expire allcoations 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-27 14:01:52 +08:00
XuanYang-cn 35656ea9ce Update pulsar client version and fix cannot close tcp connection error 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-27 13:58:52 +08:00
cai.zhang c4a58d259a Add deploy mode env variable 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-27 11:56:57 +08:00
Xiangyu Wang 5ccf0c3d70 Add log for segment row estimate 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-27 11:35:42 +08:00
xige-16 3ab8b9af25 Dlete unlocked mutex in seek function 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-27 11:16:20 +08:00
xige-16 a48d46cfaf Alter msgID’s type to bytes 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-27 09:46:54 +08:00
yukun 432b53149b Refactor msgstream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-26 20:10:11 +08:00
bigsheeper 8a4c6a039f Add search log 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-25 18:09:47 -05:00
xige-16 7e60460aee close timeTickStream in flow graph when the collection is released 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-26 18:40:04 +08:00
bigsheeper 1c274de3e0 Improve mutex in queryservice 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-25 16:09:16 -05:00
bigsheeper dfb68a7e46 Remove unused segment 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-25 16:08:08 -05:00
FluorineDog f39dcdb8f3 Support error code in segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-26 16:18:30 +08:00
neza2017 eca1e25470 Add msg id at master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-26 15:13:33 +08:00
dragondriver 41e1975611 Parallelize the processing of loading binlog and saving index files 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-26 11:19:02 +08:00
cai.zhang 30cc84b164 Add more log for indexservice and indexnode 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-26 10:01:08 +08:00
yukun 85bc2a54f0 Fix load twice hang bug 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-25 18:49:41 +08:00
bigsheeper 1e4afda824 Add mutex for query node map, search buffer and dataSyncService 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-24 15:40:59 -05:00
quicksilver 7ad9b36207 Update reviewdog/action-hadolint github action version to v1.16.1 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-25 14:41:46 +08:00
dragondriver 1b743e5c6c Parallelize the processing of SearchTask.PostExecute 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-25 10:14:09 +08:00
sunby ca3187d173 Fix data race in datanode tests 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-24 16:35:38 +08:00
sunby 972dad4d8e Add Flush segment optimization 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-24 14:43:38 +08:00
zhenshan.cao 21ffc660b0 Get rid of memory panic 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-23 19:55:00 +08:00
bigsheeper 350e0df5f7 Refactor search service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-22 16:59:46 -05:00
XuanYang-cn 51f669f1cb Flush segments concurrently & change etcd meta 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-23 18:50:13 +08:00
dragondriver 9d062b54ee Add benchmark for indexbuilder, refactor test utils 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-23 18:44:57 +08:00
zhenshan.cao 9c574b7c7b Update sdk version to 0.0.55 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-23 18:40:08 +08:00
紫晴 25f3e520f9 Add testcase for index 
Signed-off-by: 紫晴 <ting.wang@zilliz.com>
 2021-03-23 16:57:59 +08:00
bigsheeper 1634d75980 Release flowgraph properly 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-22 12:49:50 -05:00
紫晴 3be6dde459 Add smoke tag 
Signed-off-by: 紫晴 <ting.wang@zilliz.com>
 2021-03-23 14:35:30 +08:00
dragondriver 49c6eeb052 Add useful debug log in proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-22 19:28:43 +08:00
xige-16 1a1e864709 Fix query node crash when empty search 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-22 19:20:56 +08:00
zhenshan.cao c2734fa55f Fix bug and enchance system 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-22 16:36:10 +08:00
sunby f4d055cf42 Add global mutex for segment allocate 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-20 20:41:39 +08:00
XuanYang-cn 4c98f4efbf Update binlog meta proto 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-20 09:50:42 +08:00
Xiangyu Wang 9ba9bf5e8a Release memory after rocksdb.write() 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-19 20:20:32 +08:00
groot 2280791128 Add memory message stream for search 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-03-19 20:16:04 +08:00
yukun 0823382876 Refactor rmq_msgstream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-19 19:33:21 +08:00
bigsheeper ba65c38d40 Fix Deadline Exceeded 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-19 17:50:26 +08:00
dragondriver 7554b8afe7 Add default nbits when build ivf-pq index 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-19 16:37:13 +08:00
zhenshan.cao 636755827f Fix unittest bug: query node num 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-17 14:19:14 +08:00
FluorineDog 2cec04ed90 Fix empty schema proto hack 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-17 11:35:28 +08:00
yukun 207f6ae773 Fix rocksmq CreateConsumerGroup 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-16 21:37:25 +08:00
XuanYang-cn 32660d547b Add endPositions in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-16 17:55:42 +08:00
neza2017 1d105ae451 Tag test case of test_connect.py::TestConnect::test_close_repeatedly 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-16 11:19:56 +08:00
xige-16 b92ad3edd6 Fix error when search on collection without partitions 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-15 19:58:52 +08:00
zhenshan.cao ec8ad89900 Delete uncessary print 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-15 19:47:13 +08:00
neza2017 9e0fc0b169 Fix ts loop 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-15 15:45:17 +08:00
neza2017 6eb45c92eb Fix flush segment 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-13 17:05:36 +08:00
groot 47d9471498 Change gracefultime to 1 second 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-03-13 14:42:53 +08:00
sunby 7bb806b5f6 Log msgID and role when log collection 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-13 11:59:24 +08:00
Xiangyu Wang fc6472eb08 Implement RocksMQ client 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-13 11:46:50 +08:00
dragondriver dd20ab44b8 Add interface of check param util 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-13 09:57:22 +08:00
cai.zhang 012f5f360e Add init log logic for index and proxy 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-12 19:47:37 +08:00
xige-16 8a5c039137 Fix search error when running single node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-12 19:23:06 +08:00
FluorineDog e33d0a797c Migrate knowhere to segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-12 15:54:59 +08:00
godchen f3649f0419 Refactor interface and proto 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-12 14:22:09 +08:00
XuanYang-cn 7dd531d159 Fix release collection bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-11 18:42:25 +08:00
sunby 6287ffa1b4 Replace log with zap 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-11 14:15:18 +08:00
ThreadDao ca916a5c47 Add ci nightly 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-03-11 14:14:29 +08:00
dragondriver 0f2a377b62 Delete redundant insert log 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-11 11:27:42 +08:00
xige-16 d35d316056 Close seek function to pass regression test 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-11 10:21:24 +08:00
xige-16 b2b39f5720 Add field position to watchDmChannelRequest’s porto 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-11 10:08:46 +08:00
xige-16 09ae985daa Fix wrong error code in master_service_test.go 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-10 22:06:22 +08:00
neza2017 aee7c74762 Fix override index name 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-10 16:21:49 +08:00
sunby 800e31ebe4 Fix grpc startup in query node 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-10 15:49:23 +08:00
del-zhenwu b1763c3690 Disable xdist in test cases 
Signed-off-by: del-zhenwu <zhenxiang.li@zilliz.com>
 2021-03-10 15:27:26 +08:00
XuanYang-cn 8567679888 Check enableIndex before load index 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-10 14:51:00 +08:00
neza2017 7b2d67242d Fix describe segment if index not exist 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-10 14:45:35 +08:00
Xiangyu Wang c875fdf913 Add logs in rocksmq 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-10 10:02:55 +08:00
cai.zhang 60b97f9b65 Replace log of indexnode and indexservice 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-10 09:56:09 +08:00
yukun 0d499a9f99 Fix search waiting for seconds in singlenode 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-09 17:42:13 +08:00
cai.zhang 408fe6e7ad Make system return success when partitions has been already loaded 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-09 16:23:41 +08:00
dragondriver af32f442bb Split big insert message into serveral smaller 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-09 16:18:58 +08:00
FluorineDog af1900b42a Remove ConcurrentBitsetPtr in segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-09 16:16:43 +08:00
quicksilver 455cc59dbb Update publish-test-images.yaml 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-09 13:55:35 +08:00
cai.zhang aa5ecbdc02 Use the project log instead of standard log of proxyservice and proxynode 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-08 19:39:36 +08:00
sunby 2a083328ee Remove redundant mutex 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 18:04:09 +08:00
Cai Yudong c795cdbe2a Change minSegmentSizeToEnableIndex default to 1024 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-03-07 13:37:25 -06:00
sunby 76480fcefe Change default log output to stdout 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 16:24:55 +08:00
neza2017 55fc3f56aa Set grpc max recv mag as math.MaxInt32 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-08 15:49:42 +08:00
neza2017 014c4fe8ce Add config of minSegmentSizeToEnableIndex 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-08 15:46:51 +08:00
sunby ab21115d54 Init rand.Seed 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 15:25:55 +08:00
XuanYang-cn 403212c5fa Fix describe_index test cases 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-08 15:00:08 +08:00
FluorineDog ec0407d5f1 Support schema verifier 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-08 12:41:46 +08:00
sunby aeb3038545 Remove GetCount in data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 10:49:44 +08:00
ThreadDao 7a3223dc81 tags collection stats 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-03-08 10:09:48 +08:00
neza2017 3ef18e86f0 Add 0331 tag for test_partition 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-06 17:47:11 +08:00
neza2017 efa7306ef6 Remove with prefix 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-06 16:00:41 +08:00
zhenshan.cao ce5d7c7ecc Change retry times and sleep duration 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-06 11:52:43 +08:00
xige-16 f6a4bacbeb Delete field fieldIDs from ReleaseSegmentRequest’s photo 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-05 20:47:43 +08:00
dragondriver c68a8d35bf Remove timeout controlled by server, use timeout client passed instead 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-05 20:43:19 +08:00
neza2017 2835bcdc4e Update masterservice unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-05 20:41:34 +08:00
Xiangyu Wang e5d595564a Add log for singlenode 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-05 18:16:50 +08:00
yukun 33d8e13f96 Fix load-collection in singlenode 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-05 18:14:59 +08:00
quicksilver 74154a11a4 Fix deploy error during regression stage 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-05 16:52:45 +08:00
XuanYang-cn a5c6f40c60 Enable tests in test_index.py 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-05 15:21:33 +08:00
BossZou 1efcf605e2 Enlarge Proxy node max size of reveivable grpc message 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-03-05 11:16:23 +08:00
sunby ddddd65d10 Delete internal/errors package 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-05 10:15:27 +08:00
bigsheeper 447a15207e Add zap log to query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-05 09:21:35 +08:00
dragondriver 801396e3ef Disable register link logic, use proxy node directly 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-04 22:27:12 +08:00
xige-16 ebaa5d4f5b Fix error of repeated loading partition failures 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-04 19:21:44 +08:00
sunby 6dc938e6f4 Refactor code 
1. add error as return value
2. check assertion success

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-04 18:58:16 +08:00
FluorineDog ef98dab2a9 Support segcore config 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-04 17:09:48 +08:00
sunby dec0b40a66 Add log panic in log module 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-04 16:01:30 +08:00
yukun 36bae90635 Fix singlenode search timeout caused by selectcase in rmq_msgstream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-04 15:53:17 +08:00
neza2017 ba21b11e9e Fix masterservice :drop index 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-04 14:54:16 +08:00
xige-16 6cdd4d02fc Refactor seek function 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-04 14:49:51 +08:00
godchen 3ba0dc2f44 Change doc 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-04 10:35:28 +08:00
ophunter233 4a921ffd7e Stop listen to dd channel in query node 
Signed-off-by: ophunter233 <chengming.li@zilliz.com>
 2021-03-03 16:06:02 +08:00
sunby ae432436db Init start position when opening new segment 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-02 15:52:42 +08:00
XuanYang-cn c0edffc63f Fix datanode update segment statistics bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-02 15:47:37 +08:00
xige-16 795cb74c85 Fix message stream miss ack 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-02 13:15:49 +08:00
FluorineDog b7a00e30e2 Fix memory leak 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-01 11:14:51 +08:00
bigsheeper 456d0e8c23 Refactor time tick align strategy 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-27 18:33:29 +08:00