Commit Graph

1320 Commits (b46d700f9d13289263ccb154e1854b76b21c33fa)

Author SHA1 Message Date
neza2017 6019c193fc
add grpc interface to process NewSegment And FlushSegmentCompleted (#6247)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-07-01 14:58:17 +08:00
congqixia 580e3a57cf
stop add new node into cluster before validated (#6241)
* stop add new new into cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix startup & test cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-07-01 14:32:16 +08:00
neza2017 a518e408b1
add log when seek in msgstream (#6244)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-07-01 13:46:16 +08:00
FluorineDog bf8b2be4a7
Deprecate num_groups to simplify search API (#6230)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-07-01 10:32:15 +08:00
zhenshan.cao ff4c6c7ff4
Fix travel timestamp and guarantee timestamp (#6234)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-30 21:02:13 +08:00
Cai Yudong a7a9eeeffe
Add time record for build index and search process (#6231)
* add timerecord for build index process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add timerecord for query process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-30 19:46:14 +08:00
yukun 95aa3c85b6
Add index_name check for drop_index (#6217)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-30 17:56:12 +08:00
xige-16 ff93d1611f
merge retrive and search code in query node (#6227)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-30 17:50:15 +08:00
xige-16 f146d3825f
Fix search hang after querynode restart (#6212)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-30 17:48:19 +08:00
congqixia 3119ce2a6c
Stop create session in watch op (#6222)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-30 16:50:20 +08:00
godchen 49a0528d53
Fix trace (#6218)
* Fix trace

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-30 16:18:13 +08:00
Cai Yudong 6436f596de
add GetIndexStates (#6213)
* add GetIndexStates

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to index_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-30 14:32:19 +08:00
neza2017 4e4ab4adc6
check if etcd is health before register session (#6215)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-30 14:12:15 +08:00
XuanYang-cn 357af89d55
Add unittest in segment_replica (#6197)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-30 10:26:12 +08:00
congqixia ae072b4f1d
Add candidate management for datacoord dn cluster (#6196)
* Add candidate management for datacoord dn cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix test cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-30 10:20:15 +08:00
FluorineDog b90b4f2058
renaming (#6192)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-30 10:18:26 +08:00
XuanYang-cn 769cbb292a
Update doc in datanode (#6194)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-29 17:34:13 +08:00
sunby dd3ba9b2b3
Add rev by one when start watch event (#6152)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-29 14:52:14 +08:00
godchen 1c1ffc8b32
Add connect err log (#6186)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-29 14:30:17 +08:00
Cai Yudong 95135c1681
fix blob file sort error (#6184)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-29 13:24:15 +08:00
congqixia c49ce4ddc2
Add datacoord Server comment and improve para (#6182)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-29 10:46:13 +08:00
congqixia 3a202cad2b
Reduce connect timeout time (#6181)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-29 09:30:17 +08:00
yukun b87baa108a
Add topks in searchResult (#6168)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-28 15:28:11 +08:00
yukun 7df4bf8d93
Change strange expression err hint (#6128)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-28 15:18:12 +08:00
neza2017 95ad498822
check proxy if exist before connect to it (#6165)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-28 15:08:12 +08:00
dragondriver d3c503f3aa
Calculate the real topk in proxy (#6132)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-28 14:30:14 +08:00
congqixia b27e6b52bf
Force cluster refresh for each dn change event (#6161)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-28 13:28:14 +08:00
yukun ae66446f16
Fix issue-5992 (#6158)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-28 12:16:13 +08:00
Cai Yudong a9a806ace0
optimize WatchProxy (#6153)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-28 11:04:14 +08:00
yukun effcbc7cb3
Fix issue-6143 (#6145)
* Fix issue-6143

Signed-off-by: kun yu <kun.yu@zilliz.com>

* Return error when output_fields contains vector field

Signed-off-by: kun yu <kun.yu@zilliz.com>

* Fix when output_fields contain primarykey

Signed-off-by: kun yu <kun.yu@zilliz.com>

* Add primarykey

Signed-off-by: kun yu <kun.yu@zilliz.com>
2021-06-28 10:12:13 +08:00
xige-16 f973456c10
filter delEvent when querynode register fail (#6150)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-27 12:16:09 +08:00
xige-16 ca4cba152c
return loaded collection when ShowCollectionsType_InMemory (#6149)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-27 12:10:08 +08:00
xige-16 6036ef2c7d
Fix standalone can't start after down (#6148)
* mutil go routine do registerNode and  deleteNode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* stop retry create querynode client

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-26 22:28:10 +08:00
neza2017 b22ab71222
fix rmq (#6147)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-26 22:26:20 +08:00
neza2017 6d7edc6015
fix typo (#6142)
* fix typo

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add log

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-26 18:04:12 +08:00
xige-16 6b65529086
Fix search error after releasePartition (#6135)
* fix search error after releasePartition

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-26 16:08:11 +08:00
neza2017 2e37fca214
add log in session (#6136)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-26 14:14:11 +08:00
zhenshan.cao 42be531f9c
Add InsertCnt (#6131)
* Add InsertCnt

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Enhance log

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-26 14:02:12 +08:00
yukun e94a27856e
Mkdir rocksmq_path if not exist (#6134)
Signed-off-by: kun yu <kun.yu@zilliz.com>
2021-06-26 12:28:07 +08:00
xige-16 8db95a63f6
Return error when search limit equal 0 or grater than 16384 (#6115)
* return error when search limit equal 0 or grater than 16384

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-26 11:44:11 +08:00
neza2017 a1d0e5c980
remove task scheduler (#6123)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-26 09:22:11 +08:00
bigsheeper 0d0c1901dd
Fix service time check of search in streaming (#6068)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-25 20:18:09 +08:00
yukun 39614aa8eb
Add rocksmq_path in config and paramtable (#6099)
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-25 19:44:11 +08:00
congqixia 384f440dbf
Fix panic when BuildIndex 0 while err = nil (#6125)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-25 17:58:09 +08:00
yukun eb818d05ab
Remove vector fields from output fields (#6112)
* Remove vector fields from output fields

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add empty_collection error_code

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-25 16:52:11 +08:00
neza2017 da12e55c6c
rootcoord connect to other components in go routines (#6117)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-25 16:48:10 +08:00
godchen 73996aba91
Change client init timeout (#6107)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-25 15:50:23 +08:00
Cai Yudong 64d1fa2de2
Banner use UTC time (#6102)
* banner use UTC time

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-25 10:24:11 +08:00
xige-16 cea83dde5a
fix multi loadCollectition not work (#6097)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-24 21:10:13 +08:00
yukun 9b549e0fa4
Add primaryKey to output fields (#6094)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-24 21:08:14 +08:00
Cai Yudong f633d48ccc
fix rootcoord unittest (#6090)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-24 19:44:08 +08:00
groot a807dca132
Fix #6082 Unstable issue for standalone mode in CI (#6084)
* Fix #6082 Unstable issue for standalone mode in CI

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Fix #6082 Unstable issue for standalone mode in CI

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-06-24 19:28:07 +08:00
bigsheeper e87372761a
Remove tSafe in release partition task and return max timestamp (#6078)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-24 19:06:34 +08:00
sunby 25d3fead9f
Change error type to error functions in data coord (#6076)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-24 19:05:42 +08:00
godchen 3785bb6330
Change retry times (#6083)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-24 19:05:06 +08:00
congqixia 10e5fc3382
Fix query node crash when retrieved 0 result (#6087)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-24 19:04:34 +08:00
yukun 0f409da136
Fix standalone seek hang after restart (#6073)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-24 16:44:07 +08:00
xige-16 710e2ca185
return error before queryCoord ready (#6067)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-24 16:00:15 +08:00
dragondriver 133814be8e
Fix pq params check when creating index (#6052)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-24 15:40:10 +08:00
godchen 064f8f76cd
change_coord_connect_retry (#6064)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-24 15:20:08 +08:00
neza2017 bf068d84ad
fix index coord (#6042)
* fix index coord

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove memstream from RmsFactory

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-24 14:36:08 +08:00
congqixia f6761bbc33
Refactor datacoord segment status (#6039)
* Refactor datacoord segment status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove lastExpireTime

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-24 14:20:10 +08:00
yukun c8bf9078ae
Change pchannels to vchannel in retrieveResults (#6054)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-24 13:40:03 +08:00
yukun b72e4c6372
Merge search_collection and retrieve_colletion into query_collection (#6037)
* Merge search_collection and retrieve_colletion into query_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix static-check

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-23 20:26:10 +08:00
bigsheeper dca24f313a
fix search timeout error (#6027)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-23 18:04:22 +08:00
bigsheeper 0f89f40488
Use virtual channel in search result and retrieve result (#6030)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-23 18:00:37 +08:00
xige-16 5417fee45f
fix releasePartition not work after loadCollection (#6025)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-23 17:44:12 +08:00
dragondriver 1f524bdf3e
Fix id allocator (#6024)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-23 17:34:05 +08:00
bigsheeper 47d89abaa0
get channels by static converting (#6015)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-23 17:26:09 +08:00
godchen 17303bf4d3
Change session doc (#6022)
* change session doc

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-23 17:14:10 +08:00
sunby 46f86a0354
Return sealed segments when calling flush (#6011)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-23 16:56:11 +08:00
Cai Yudong a09a3a1905
rename service to coord (#6020)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-23 16:14:08 +08:00
neza2017 9ebca59099
fix time tick (#6017)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-23 15:28:09 +08:00
congqixia d6259d2dd6
Refactor policy (#6003)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-23 14:28:08 +08:00
XuanYang-cn b640627a38
Fix datanode restart bug (#5976) (#6004)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-23 12:26:10 +08:00
sunby a529410b38
Fix data race in internal/datacoord/meta.go (#6005)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-23 12:20:06 +08:00
sunby fe8432016d
Refactor data coordinator (#6008)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-23 12:10:12 +08:00
dragondriver a493151f70
Fix #5951 (#6010)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-23 12:06:10 +08:00
godchen fd07a44dba
Fix error in retry refactor (#6000)
* Fix error in retry refactor

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-23 11:48:06 +08:00
Cai Yudong ba5aacd3f2
Remove GetChanNameByVirtualChan (#5999)
* remove redundant ctx

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* remove GetChanNameByVirtualChan

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-23 11:18:05 +08:00
godchen 99be4c09a5
change retry (#5996)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-23 09:24:10 +08:00
Cai Yudong 7dbf33b788
Update component names in config files (#5989)
* rename master/indexService in .yaml

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proxyNode to proxy in config files

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move master.yaml to root_coord.yaml, move proxy_node.yaml to proxy.yaml

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move proto/query_service.proto to proto/query_coord.proto

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move proxy_node_manager.go to proxy_manager.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add query_coord.pb.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 19:08:03 +08:00
yukun 792dba4ae3
Fix id field insert failed when auto_id is true (#5984)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-22 18:46:04 +08:00
FluorineDog b51893d3f1
Enrich retrieve task error message (#5975)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-22 18:42:03 +08:00
dragondriver fbaa847641
Add metrics for proxy (#5987)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-22 18:32:08 +08:00
sunby 0978b93a21
Refactor data coordinator (#5982)
Rename variable name and make error msg more clear

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 18:24:08 +08:00
zhenshan.cao 0f05622c09
Modify error msg when index is not created (#5978)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-22 18:04:07 +08:00
congqixia 7a2546c24a
Fix proxy panic if quit before init (#5981)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-22 17:40:07 +08:00
bigsheeper b31bae351d
fix empty search result (#5979)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-22 16:52:08 +08:00
xige-16 2af1c27811
Rename queryservice to querycoordinator (#5931)
* Rename queryservice to querycoordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix param table

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix unittest

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* service to coordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* set querycoord in ci

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-22 16:44:09 +08:00
yukun 282d2624e0
Not remove rocksdb folder when close and init (#5974)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-22 16:20:10 +08:00
sunby d2c8462dac
Rename params in data_coord.yaml (#5968)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 16:18:21 +08:00
Cai Yudong d5f4ee6f44
Rename proto master/index_service/proxy_service (#5973)
* rename proto proxy_service to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto index_service to index_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto master to root_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 16:14:09 +08:00
neza2017 80bb418136
Release partition (#5971)
* add debug info

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* notify query coord to release partitions after drop partition

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-22 16:08:08 +08:00
dragondriver fd57554d32
Add index params check in proxy (#5958)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-22 15:28:04 +08:00
Cai Yudong 2926a78968
Rename proxynode to proxy (#5961)
* rename package name from proxynode to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move distributed/proxynode to distributed/proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move internal/proxynode to internal/proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proxynode to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 14:40:07 +08:00
xige-16 7039fb7c82
Fix the problem of stuck after loadBalance and loadFieldData error (#5960)
* delete nodeInfo after nodeDown

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance can's stop

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load field data error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* contiue task loop after error in queryService

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-22 14:10:09 +08:00
neza2017 5cdc1af1af
fix proxy node (#5965)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-22 14:04:08 +08:00
sunby c863a193de
Rename filename from dataservice to datacoordinator (#5946)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 10:42:07 +08:00
congqixia c8420cfa10
Add proxy healthy check for working services (#5954)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-22 10:36:08 +08:00