milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,915 Commits
136 Branches
115 Tags
566 MiB
Commit Graph

457 Commits (c3e504b3fa69289f49397bbfd0f617b619ca62ab)

Author SHA1 Message Date
xige-16 b035ec5c7b
Enable search when querynode only load segment (#7104) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-08-16 12:24:09 +08:00
Cai Yudong e771bda92f
optimize retrieve output vector code structure (#7102) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-14 11:18:10 +08:00
XuanYang-cn 6c5be83ea1
Refactor NewEtcdKV API (#7075) 
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 11:04:09 +08:00
Xiangyu Wang 5e78fd6a97
[skip ci]Update approvers (#7063) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-12 19:36:09 +08:00
Cai Yudong 6c75301c70
optimize search reduce logic (#7066) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-12 18:00:11 +08:00
congqixia 3f21e3d062
Fix querynode tt producer leak (#7059) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-12 14:16:08 +08:00
bigsheeper b9d3897deb
Return error when any failed detected in loadSegment (#7005) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-12 14:14:08 +08:00
godchen 7fc7623b92
Add vector data memroy existence check (#7044) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-11 19:50:09 +08:00
bigsheeper 4123deef9e
Return error message in grpc.LoadSegment in query node (#6908) 
* return load error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* init streaming meta in loadSegment

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix excluded init

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* improve impl

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-08-03 22:01:27 +08:00
Xiaofan aba234fa3f
Fix Msgstream exponential retry (#6807) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-03 10:39:24 +08:00
Cai Yudong fe50f97726
Use id to tell search result validation in proxy reduce stage (#6905) 
* use id to tell search result validation

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable test_search_binary_hamming_flat_index

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix merge retrieve result issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-02 10:25:49 +08:00
bigsheeper e8ea8b51d5
improve query node log (#6897) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-31 10:47:22 +08:00
bigsheeper eee0f1e077
Add mutex for segment of segCore in querynode (#6870) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-29 16:03:22 +08:00
godchen db94d7771f
Read vector from disk (#6707) 
* Read vector from disk

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix git action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix test error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix caculate error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change var name

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix len error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused code

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change bytes to float method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change float to bytes method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-07-24 09:25:22 +08:00
bigsheeper 21bc5810c0
remove redundant log of query node (#6768) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-23 20:09:33 +08:00
bigsheeper 2f44fc40a3
fix historical unittest (#6705) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-23 19:55:07 +08:00
Cai Yudong cbb4052fdb
Support search and query output fields using wildcard (#6671) 
* update wildcard polocy for search and query

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* mark xfail cases

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* mark xfail cases

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize debug log

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-21 14:52:12 +08:00
FluorineDog 234954931f
remove deprecated (#6623) 
* remove deprecated

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* fix timeout

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-20 15:18:08 +08:00
Cai Yudong a7b27db63e
update fillRetrieveResults parameter (#6598) 
* update fillRetrieveResults parameter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize fillVectorFieldData process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-17 15:17:30 +08:00
Cai Yudong a992dcf6a8
Support query return vector output field (#6570) 
* improve code readibility

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add offset in RetrieveResults

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add VectorFieldInfo into Segment struct

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add new interface for query vector

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update load vector field logic

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update load vector field logic

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fill in field name in query result

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add FieldId into FieldData

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add fillVectorOutputFieldsIfNeeded

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update data_codec_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add DeserializeFieldData

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* realize query return vector output field

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* disable query vector case

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-16 17:19:55 +08:00
Cai Yudong 1c29b42434
change output_fields to out_fields_id for RetrieveRequest (#6506) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-14 13:11:54 +08:00
bigsheeper e50d39dfe1
Query result get global sealed segments from ETCD (#6377) 
* get global sealed segments from ETCD

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix race error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-14 11:15:54 +08:00
Cai Yudong 724f10b9a0
Unify the usage of query and search (#6467) 
Unify the usage of query and search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-13 22:20:33 +08:00
xige-16 2b720fd2f7
Fix load hang when restart querynode many time in short time (#6351) 
* fix load hang when restart querynode many time in short time

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add multi queryNode ut

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add ut for restart querynode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* set queryCoord contex to load collection

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-07-13 14:16:00 +08:00
dragondriver bd317a5461
Check if field data is empty (#6404) 
* Check if field data is empty

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* FixBug: fast fail when output_fields contain fields that are not in schema

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-07-11 19:39:52 +08:00
neza2017 8b89744c6b
search with exp (#6409) 
* process float vector and binary vector field
on the search result

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add more debug info on etcd

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add debug info when grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-07-10 10:21:52 +08:00
Cai Yudong 21a9bcd844
Create default partition when receive CreateCollectionMsg (#6305) 
* create default partition when create collection

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* adjust sleep time for unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-06 09:16:03 +08:00
FluorineDog bf8b2be4a7
Deprecate num_groups to simplify search API (#6230) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-01 10:32:15 +08:00
zhenshan.cao ff4c6c7ff4
Fix travel timestamp and guarantee timestamp (#6234) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-30 21:02:13 +08:00
Cai Yudong a7a9eeeffe
Add time record for build index and search process (#6231) 
* add timerecord for build index process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add timerecord for query process

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-30 19:46:14 +08:00
xige-16 ff93d1611f
merge retrive and search code in query node (#6227) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-30 17:50:15 +08:00
xige-16 f146d3825f
Fix search hang after querynode restart (#6212) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-30 17:48:19 +08:00
Cai Yudong 95135c1681
fix blob file sort error (#6184) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-29 13:24:15 +08:00
xige-16 6b65529086
Fix search error after releasePartition (#6135) 
* fix search error after releasePartition

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-26 16:08:11 +08:00
xige-16 8db95a63f6
Return error when search limit equal 0 or grater than 16384 (#6115) 
* return error when search limit equal 0 or grater than 16384

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-26 11:44:11 +08:00
bigsheeper 0d0c1901dd
Fix service time check of search in streaming (#6068) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-25 20:18:09 +08:00
yukun 39614aa8eb
Add rocksmq_path in config and paramtable (#6099) 
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 19:44:11 +08:00
yukun eb818d05ab
Remove vector fields from output fields (#6112) 
* Remove vector fields from output fields

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add empty_collection error_code

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 16:52:11 +08:00
bigsheeper e87372761a
Remove tSafe in release partition task and return max timestamp (#6078) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-24 19:06:34 +08:00
congqixia 10e5fc3382
Fix query node crash when retrieved 0 result (#6087) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-24 19:04:34 +08:00
yukun c8bf9078ae
Change pchannels to vchannel in retrieveResults (#6054) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-24 13:40:03 +08:00
yukun b72e4c6372
Merge search_collection and retrieve_colletion into query_collection (#6037) 
* Merge search_collection and retrieve_colletion into query_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix static-check

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-23 20:26:10 +08:00
bigsheeper dca24f313a
fix search timeout error (#6027) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-23 18:04:22 +08:00
bigsheeper 0f89f40488
Use virtual channel in search result and retrieve result (#6030) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-23 18:00:37 +08:00
xige-16 5417fee45f
fix releasePartition not work after loadCollection (#6025) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-23 17:44:12 +08:00
bigsheeper 47d89abaa0
get channels by static converting (#6015) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-23 17:26:09 +08:00
Cai Yudong a09a3a1905
rename service to coord (#6020) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-23 16:14:08 +08:00
godchen fd07a44dba
Fix error in retry refactor (#6000) 
* Fix error in retry refactor

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 11:48:06 +08:00
godchen 99be4c09a5
change retry (#5996) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 09:24:10 +08:00
bigsheeper b31bae351d
fix empty search result (#5979) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-22 16:52:08 +08:00
xige-16 2af1c27811
Rename queryservice to querycoordinator (#5931) 
* Rename queryservice to querycoordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix param table

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix unittest

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* service to coordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* set querycoord in ci

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-22 16:44:09 +08:00
xige-16 7039fb7c82
Fix the problem of stuck after loadBalance and loadFieldData error (#5960) 
* delete nodeInfo after nodeDown

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance can's stop

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load field data error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* contiue task loop after error in queryService

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-22 14:10:09 +08:00
yukun b2eb596c23
Change search result to columnBased and return target_entry (#5896) 
* Change search result to columnBased and return target_entry

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: fluorinedog <fluorinedog@gmail.com>

* synchronize the latest pymilvus-orm version

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change pymilvus-orm version to 40

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-21 20:18:13 +08:00
bigsheeper 2e615907ca
fix test_search_after_index_different_metric_type (#5921) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-21 18:46:15 +08:00
sunby 7f2d7c1da3
Rename dataservice to datacoordinator (#5912) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 18:22:13 +08:00
Cai Yudong 92e429d812
Rename IndexService to IndexCoord (#5932) 
* rename package indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix queryservice static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move distributed/indexservice to distributed/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move internal/indexservice to internal/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename MasterComponent to RootCoordComponent

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for queryservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for dataservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for datanode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for proxynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for querynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix rebase issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-21 17:28:03 +08:00
zhenshan.cao 958280ebb8
Add support for time travel (#5897) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-21 10:42:10 +08:00
bigsheeper bbd8a7e13a
Add released partitions and fix search error in empty partition (#5893) 
* rename flowgraphType to loadType

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* add load type and released partitions

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* filter released partition

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-19 18:38:07 +08:00
FluorineDog 255e3959af
support time travel (#5894) 
* support time travel

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-19 17:38:11 +08:00
XuanYang-cn ab7f642740
Add alias in paramtable (#5878) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-19 12:38:06 +08:00
xige-16 e4c51aae36
Add recovery logic for querynode and queryservice (#5843) 
* merge milvus/recovery2

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add recovery logic in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug smoke case

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add etcd to querynode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix release partition error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug querynode down and recovery

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add log

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix showCollection

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* skip smoke test search without insert

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-19 11:45:09 +08:00
sunby 33f5225968
Use physical channel to init msg stream (#5815) 
* Use physical channel to init msg stream

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix search collection error check (#1)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: bigsheeper <954206947@qq.com>
 2021-06-18 16:02:05 +08:00
bigsheeper 46151d203f
Fix tSafe cannot update correctly (#5820) 
* Fix tSafe cannot update correctly

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* skip test_load_partitions_release_collection

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* skip test_release_collection_during_searching

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-17 16:56:04 +08:00
yukun e9a8d1c404
Add vChannels in proxy for query results (#5802) 
* Fix proxynode for new retrieve logic

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove querynodenum from proxynode and querynode

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove QueryNodeIDList from proxy

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-16 20:15:59 +08:00
bigsheeper d6f7fbf19f
release collection synchronously (#5798) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-16 15:31:57 +08:00
bigsheeper bf39644457
Fix channels in query node and fix search (#5795) 
* fix add channel error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix tSafe vChannel key

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-16 12:05:56 +08:00
xige-16 97049ce293
use dataservice's getRecoveryInfo to get load and watch info (#5790) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-16 11:09:56 +08:00
yukun 5c18138a6b
Fix retrieve_collection return nil vchannel (#5787) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-15 20:07:30 +08:00
bigsheeper 58bf92a407
Add vChannel,pChannel in query node and get vChannel from master in query service (#5784) 
* add vChannel,pChannel in query node and get vChannel from master in query service

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-15 20:06:10 +08:00
yukun fceba63a33 Add segment and channel msgs in retrieve results (#5750) 
* Fix query hang bug (#5709)

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add segment and channel msgs in retrieve results

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-15 16:38:59 +08:00
zhenshan.cao 5c69c6902d Refactor proxy channelTimeTick and fix search timeout (#5760) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
zhenshan.cao 33a43f48c8 [skip ci] Fix bug: Search timeout and refactor client init logic (#5735) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:24:48 +08:00
xige-16 f2e551e9a0 Add asConsumer before seek (#5706) 
* add asConsumer before seek

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* get dmChannel from master

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* use pChannel instead in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-15 16:14:27 +08:00
FluorineDog bec9f2c182
Split segcore and plan proto for future feature (#5767) 
* Split segcore and plan proto for future feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-15 14:43:57 +08:00
bigsheeper cdbc6d2c94
Refactor query node and query service (#5751) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 12:41:40 +08:00
yukun e97936ba1c
Fix query hang bug (#5709) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-10 14:33:49 +08:00
bigsheeper 8aae0f7cc9
Refactor flow graph and load/watchDML in query node (#5682) 
* Refactor flow graph and load/watchDML in query node

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-09 11:37:55 +08:00
zhenshan.cao bc46e4780d
Fixbug:index is not loaded in some cases (#5633) 
* Fixbug:index is not created in some cases

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Delete unecessary print

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* fix create index

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-06 09:41:35 +08:00
zhenshan.cao 7dac20c35c
Add log for poxynode transform data and querynode segment insert (#5627) 
* Add log for poxynode transform data and querynode segment insert

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Decrease print times

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-05 16:21:36 +08:00
zhenshan.cao 03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
FluorineDog 9a90313390
Support GetEntityByIDs in CGo, fix segcore bugs (#5563) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-04 10:38:34 +08:00
Xiangyu Wang e80f06c030
Add retrieve process log (#5561) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:56:33 +08:00
Xiangyu Wang f791ccb246 Update OWNERS file 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:01:24 +08:00
yukun 57ece4a771
Fix retrieve_collection bugs (#5575) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-03 19:41:33 +08:00
zhenshan.cao b0b8f58192
Fix bug: search timeout (#5557) 
* Fix bug: search timeout

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Add log and fix unittest bug

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-03 14:58:34 +08:00
yukun dcb4161c9f
Fix retrieve_service (#5531) 
* Fix retrieve_service

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix retrieve_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-02 19:18:33 +08:00
yukun 48f35ffa97
Add publishRetrieveResult (#5501) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-31 18:08:32 +08:00
Xiangyu Wang ca0993c274 Update OWNERS 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-30 18:58:47 +08:00
Cai Yudong b414800d49
Update Seek interface (#5492) 
* update Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Seek for mqTtMsgStream

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use Retry in Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-29 23:21:34 +08:00
yukun cd549f812d
Add retrieve_service (#5481) 
* Add retrieve_service

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix for tSafeReplica

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-29 11:16:29 +08:00
bigsheeper 03f90ff28c
refactor collection replica and add tsafe replica in querynode (#5475) 
* refactor collection replica and add tsafe replica in querynode

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* static check

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* static check2

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* code format

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-05-28 15:40:32 +08:00
xige-16 fce792b8bf
Add historical and streaming module in querynode (#5469) 
* add historical and streaming

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix GetSegmentInfo

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* pass regression test

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2021-05-28 10:26:30 +08:00
neza2017 f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
neza2017 c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00
zhenshan.cao 3e73775cc1
Add ComputeProduceChannelIndexes for msgstream (#5364) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-05-24 08:30:09 +00:00
neza2017 e326eaad1e
master client,reconnect if grpc failed (#5360) 
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-24 01:44:49 +00:00
godchen 0eee7e5252
Change session (#5329) 
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-21 11:28:52 +00:00
sunby c1efa53779
Remove redundant fields in SegmentInfo and related structure (#5280) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-21 06:51:25 +00:00
Xiangyu Wang 23c4de0eb8
Flush statistics for all int64 fields (#5318) 
Resolves: #5262

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-20 10:38:45 +00:00
godchen e224d1e725
Add SessionManager (#5288) 
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-19 10:36:05 +00:00
godchen b74afd7a0e
Add service registration (#5189) 
Add service registration.
Part of Issue #5174.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-14 02:05:18 +00:00
dragondriver b7977698a8
Change the codec style of expr plan from text to binary (#5129) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 11:27:17 +00:00
dragondriver e5d4963ba0
Distinguish the dsl and expr in proxy and query node (#5118) 
Pass a parameter `IsExpr` from proxy to query node, so query 
node can create search plan according to the variable `IsExpr`.
It makes `segcore` unnecessary to distinguish `dsl` or `expr`
using dynamic exception. When `IsExpr` is set to true, query
node will call `CreatePlanByExpr` according to the passed
information about expression. Otherwise query node will keep
still to use `CreatePlan` according to the `dsl` information. At
the same time, this pr adds some unittests to `CreatePlanByExpr`,
these unittests translate already exist case with `dsl` to case with
`expr`.

Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 07:20:47 +00:00
FluorineDog 88f5642603
Add plan proto and support basic boolean expr parser (#5088) 
**What type of PR is this?**
- [x] Feature

**What this PR does / why we need it:**
This PR supports boolean expression as DSL.
1. The goal of this PR is to support predicates
    like `A > 3 && not B < 5 or C in [1, 2, 3]`. 
2. Defines `plan.proto`, as Intermediate Representation (IR) 
    used between go and cpp. 
3. Support expr parser, convert predicate expr to IR
    in proxynode, while doing static check there
4. Support IR to AST in cpp, enable the execution
 2021-04-29 08:48:06 +00:00
bigsheeper 88e1d651cb
Remove partition and segment ids properly in query node (#5090) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

**What type of PR is this?**

- [ ] API-change
- [x] BUG
- [ ] Improvement
- [ ] Documentation
- [ ] Feature
- [ ] Test and CI
- [ ] Code Refactoring

**Which issue(s) this PR fixes:**

Fixes #

**What this PR does / why we need it:**

Not Available
 2021-04-29 07:59:08 +00:00
bigsheeper 4197b094bf
Use internal dsl and placeholder group (#5033) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-25 06:08:28 +00:00
bigsheeper c4b379be8b
Set search graceful time from config (#5014) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-24 07:53:37 +00:00
bigsheeper 2b5bedf736
Use multiple search channels (#5018) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-24 02:44:28 +00:00
bigsheeper 00bb1014cd
Add search channels to query node (#5012) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-23 10:11:26 +00:00
bigsheeper dadb02db75
Remove unused params in query node (#4987) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-23 10:07:45 +08:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
sunby fad94fb576
Remove unsed configs (#4990) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-22 03:04:59 +00:00
xige-16 1165db75f6 Optimize search performance 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-19 19:30:36 +08:00
bigsheeper 71b8f309b2 Add licence to query node and query service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-19 13:47:10 +08:00
bigsheeper 08c5134286 Fix collection cannot be found when searching 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-16 17:37:50 +08:00
bigsheeper 9e7559b865 Make query node asynchronously load and release collection 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-16 14:40:33 +08:00
FluorineDog ce7a5ea699 Support segcoreinit 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-16 14:02:49 +08:00
XuanYang-cn e281f6a4a1 Remove useless configs in config files 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-13 10:04:39 +08:00
godchen deba964590 Change name of kvBase 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-12 18:09:28 +08:00
bigsheeper 39458697c7 Use asynchronous functions of load and release 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-12 09:18:43 +08:00
xige-16 22eda19bc4 Fix mem leak when load index 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-10 10:10:53 +08:00
sunby 751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
xige-16 332b14c915 Delete querynode’s redundant load IO 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-04-07 18:29:19 +08:00
bigsheeper fe3fa8dd1d Fix search service unittest 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-01 15:39:23 -05:00
Xiangyu Wang 87a1e0b662 Reorganize msgstream 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-02 13:48:25 +08:00
xige-16 c10b36f188 fix getting wrong offset when segment preinsert 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-31 16:16:58 +08:00
xige-16 a6f1de036b Optimize search performance in query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-30 22:16:58 +08:00
zhenshan.cao efbf4452c5 Add log for port changed or proto unmarshal error 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-30 20:19:30 +08:00
godchen 8f58d5641b Perfect trace and add trace init 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-30 16:36:34 +08:00
Xiangyu Wang 8d66388b8c Add logs for querynode 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-30 15:00:14 +08:00
yukun 432b53149b Refactor msgstream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-03-26 20:10:11 +08:00
bigsheeper 8a4c6a039f Add search log 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-25 18:09:47 -05:00
xige-16 7e60460aee close timeTickStream in flow graph when the collection is released 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-26 18:40:04 +08:00
bigsheeper dfb68a7e46 Remove unused segment 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-25 16:08:08 -05:00
FluorineDog f39dcdb8f3 Support error code in segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-26 16:18:30 +08:00
neza2017 eca1e25470 Add msg id at master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-26 15:13:33 +08:00
bigsheeper 1e4afda824 Add mutex for query node map, search buffer and dataSyncService 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-24 15:40:59 -05:00
quicksilver 7ad9b36207 Update reviewdog/action-hadolint github action version to v1.16.1 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-25 14:41:46 +08:00
bigsheeper 350e0df5f7 Refactor search service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-22 16:59:46 -05:00
bigsheeper 1634d75980 Release flowgraph properly 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-22 12:49:50 -05:00
xige-16 1a1e864709 Fix query node crash when empty search 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-22 19:20:56 +08:00
zhenshan.cao c2734fa55f Fix bug and enchance system 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-22 16:36:10 +08:00
groot 2280791128 Add memory message stream for search 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-03-19 20:16:04 +08:00
zhenshan.cao 636755827f Fix unittest bug: query node num 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-17 14:19:14 +08:00
FluorineDog 2cec04ed90 Fix empty schema proto hack 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-17 11:35:28 +08:00
xige-16 b92ad3edd6 Fix error when search on collection without partitions 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-15 19:58:52 +08:00
groot 47d9471498 Change gracefultime to 1 second 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-03-13 14:42:53 +08:00
xige-16 8a5c039137 Fix search error when running single node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-12 19:23:06 +08:00
godchen f3649f0419 Refactor interface and proto 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-12 14:22:09 +08:00