milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
16,083 Commits
127 Branches
113 Tags
553 MiB
Commit Graph

1225 Commits (633a74988091f8890c5d4734437e4e6b1263e7c1)

Author SHA1 Message Date
Jiquan Long 875b6f88b0
Add channel information to flow graph node (#17349) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-15 21:04:10 +08:00
xige-16 98b32f3eba
Improve time record log in search/query (#17545) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-15 20:48:10 +08:00
xige-16 dd483beefd
Delete the log that prints a lot of deleteMsg.timestamps (#17543) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-15 20:46:14 +08:00
xige-16 16f0815184
Add time record log on search/query execution path (#17497) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-13 11:42:09 +08:00
yah01 a38790f586
Predict the loading memory usage more accurate (#17451) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-09 18:16:07 +08:00
yah01 70f8bea4b4
Avoid growing slice as deserializing binlogs (#17421) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-08 11:46:06 +08:00
zhenshan.cao 98e95275fe
Fix several code issues caused by carelessness (#17385) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-06 18:58:05 +08:00
congqixia ac6394d0fd
Fix container/list removal in iteration (#17382) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-06 16:34:05 +08:00
Xiaofan af994b5e1c
Fix etcd watch error (#17364) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-06 16:26:06 +08:00
bigsheeper 45febac298
Extract lease of cProto in QueryNode (#17373) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 15:56:05 +08:00
Ten Thousand Leaves f5e63177d2
Fix an issue with bulk load with multiple DataNodes (#17324) 
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-02 18:54:04 +08:00
yah01 4e83f37899
Improve load performance (#17273) 
Load segment's fields concurrently

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-02 17:56:03 +08:00
xige-16 8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel (#17308) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-02 16:56:04 +08:00
zhenshan.cao 5f44e45480
Add metrics on QueryNode and Proxy (#17328) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 16:06:03 +08:00
bigsheeper 7a59a80506
Remove segmentPtr lock and use collection lock instead (#17303) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 14:00:04 +08:00
congqixia c76b4ade01
Fix data race in LoadSegmentsTask (#17342) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-02 12:52:03 +08:00
zhenshan.cao 2763efc9b0
Add cache of grpc client of ShardLeader in proxy (#17301) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 12:16:03 +08:00
xige-16 63a31ccb38
Remove error return value when removeSegment (#17310) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-01 13:18:02 +08:00
bigsheeper f55fe6be3c
Merge historical replica and streaming replica in QueryNode (#17217) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-31 13:42:03 +08:00
congqixia c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster (#16856) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:50:04 +08:00
congqixia cecbd40a22
Fix Data race of loadType in querynode/collection.go (#17275) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:16:02 +08:00
bigsheeper 11834a0948
Improve query log in QueryNode (#17268) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-30 17:18:03 +08:00
Jiquan Long ac2cadc83b
Refine segment type logs to string (#17248) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-30 15:30:01 +08:00
zhenshan.cao ad9276a440
Add metrics for proxy and querynode (#17205) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-27 14:12:01 +08:00
xige-16 e196caf604
Force reclamation of memory after loading (#17244) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-27 12:50:00 +08:00
xige-16 56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
zhenshan.cao ec1103ca27
Simplify the merge logic of searchTask (#17194) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 21:27:59 +08:00
bigsheeper a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph (#17096) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-24 21:11:59 +08:00
zhenshan.cao d05c24a0ff
Delete unused code (#17182) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 14:49:59 +08:00
zhenshan.cao 2512e668f7
Update timestmap according to gracefultime for Bounded Consistency (#17171) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 12:05:59 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
Letian Jiang 70825a35cf
Make querynode LoadSegments interface idempotent (#17109) 
This PR makes the following changes:
* separate LoadSegmentsTask into two phases: PreExecute and Execute
* filters out segments that are already loaded in PreExecute phase

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-20 09:15:57 +08:00
congqixia ae2864639e
Fix replica freeAll order (#17106) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 14:53:56 +08:00
congqixia 7eb6fcfb34
Limit the query node timetick log rate (#17065) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 10:19:56 +08:00
congqixia b95681770b
Add deferred clean up for ShardCluster (#17094) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-18 21:51:56 +08:00
Jiquan Long 99b4042f8c
Disable segment small index (#16935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-18 15:27:56 +08:00
congqixia 5282397c1b
Fix VectorChunkManager cache enable parameter (#17058) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-17 17:37:55 +08:00
xige-16 c2f1d65fa1
Print err info when insert msg not aligned (#16305) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-17 13:01:57 +08:00
congqixia ffff291018
Fix Handoff log in shardCluster service (#17039) 
Fix #17035

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-16 18:29:56 +08:00
congqixia 94f0951f0d
Fix query lock logic in query shard (#17034) 
Previously query shard locks the querylock in collectionReplica before any search/query
The lock range is too large and easy to cause dead lock

This PR makes following changes:
- Rename collectionReplica to metaReplica which is more reasonable
- Make release collection operation cancels waiting search/query request
- Reduce the queryLock to collection level
- Add some unit tests for timeout & released case

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-16 18:23:55 +08:00
Xiaofan 338edcd3cc
Load binlog for different field in parallel (#17005) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-15 16:11:54 +08:00
congqixia 23f7ef931c
Remove not used QueryService (#16973) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-13 20:27:54 +08:00
congqixia ec66ac69ac
Simplify chunk manager in QueryShard (#16976) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-13 19:55:54 +08:00
Letian Jiang f0dee539bb
Improve query shard log (#16950) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-13 13:45:53 +08:00
XuanYang-cn 43b36b4e5f
Make proxy update cache with NotShardLeader status code (#16940) 
See also: #16926

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-13 11:35:53 +08:00
Jiquan Long c5149c7e85
Replace text codec of pb with binary codec (#16955) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-13 09:53:53 +08:00
Letian Jiang b3eb2b1d0d
Support deltaLog loading on growing segment (#16903) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-12 11:59:53 +08:00
congqixia a8b81e215b
Fix deadlock in release replica (#16925) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-11 21:57:53 +08:00
congqixia a0614a3ff8
Use NodeIds instead of NodeID in segment info (#16912) 
The NodeID field is deprecated in current design
Query Node segment detector shall read NodeIds instead of NodeID

Fix #16911

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-11 17:39:53 +08:00
congqixia 34833680da
Add lock before query release collection from replica (#16880) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-10 18:09:53 +08:00
Ten Thousand Leaves 216c45fbd6
Add tests where segments load delta logs (#16825) 
issue: #16821

/kind improvements

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-10 17:59:52 +08:00
congqixia c7935d1a86
Add legacySegments for load balance segments rc track (#16831) 
- Add legacySegments in ShardCluster tracking old segment inUse
- Apply node id check for segment online/inUse check
- Add unit test simulating load balance case

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-07 18:25:52 +08:00
Xiaofan 92b6293be4
Fix QueryNode log level (#16604) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-07 10:27:51 +08:00
Letian Jiang b0053b7a45
Add isShardLeader field in search/query request (#16798) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-06 16:55:51 +08:00
congqixia 8c004fa981
Fix replicaID is allow zero from watch channel task (#16788) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-05 18:11:51 +08:00
Xiaofan 337ad53613
Support config small index nprobe and nlist (#16731) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-05 09:49:50 +08:00
Cai Yudong 6a62ff18bf
Support easylogging config for segcore and knowhere (#16751) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-05-03 08:39:49 +08:00
bigsheeper 1a29337e6b
Print unflushed segmentIDs instead of whole SegmentInfo (#16301) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-29 19:13:47 +08:00
bigsheeper f18b3c44ee
Print msgID instead of msg (#16268) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-29 19:11:47 +08:00
congqixia 9e09434a4b
Remove not used QueryService in querynode (#16735) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-29 18:15:48 +08:00
zhenshan.cao 97757405de
Fix bug: wrong metrics name and forget to register metrics (#16714) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-29 15:35:47 +08:00
yah01 415a0e5978
Fix miss segments/DmChannel when query by nodeID (#16719) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-29 14:11:47 +08:00
Letian Jiang a38c84e7e7
Lock meta-replica during search/query (#16703) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-29 13:37:47 +08:00
xige-16 515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
jaime 68b1b82faf
Remove DataKV interface (#16692) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-28 21:03:47 +08:00
jaime 2205b878b1
Refine log in query_shard.go (#16693) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-28 17:13:48 +08:00
zhenshan.cao 0a953948af
Simplify monitoring metrics (#16687) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-27 23:03:47 +08:00
congqixia 3a6db2faeb
Fix handling segment change logic (#16695) 
Dispatch segmentChangeInfo to ShardCluster leader
Hold segment remove before search is done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-27 22:23:46 +08:00
Letian Jiang 2879796b42
Validate partitionIDs & segmentIDs in query request (#16684) 
* check partitions if they are released or unloaded
    * check segments if their collection/partition(s) are released or unloaded

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-27 19:23:48 +08:00
congqixia 0bf176a7e1
Add SyncReplicaSegments interface for QueryNode (#16672) 
Add SyncReplicaSegments for LoadCollection/LoadPartition child task
This API notifies related ShardCluster to record segment Loaded
Preventing the Etcd events is delayed, which will cause some segment not search

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-27 10:41:46 +08:00
Xiaofan b657b10eb5
Remove useless log (#16624) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-27 10:01:47 +08:00
Letian Jiang f75dedb317
Validate partitionIDs & segmentIDs in search request (#16639) 
* check partitions if they are released or unloaded
* check segments if their collection/partition(s) are released or unloaded

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-26 14:19:45 +08:00
yah01 80ae6de323
Add comments for concurrency package (#16654) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-26 11:33:45 +08:00
congqixia bb69e09736
Use fmt.Errorf instead of string concat in querynode/segment.go (#16422) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-25 16:11:48 +08:00
yah01 7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment (#16514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-25 15:57:47 +08:00
congqixia b99b65c26e
Add segment reference count and handles change info in ShardCluster (#16620) 
Resolves #16619
Add reference count for each search/query request
For SegmentChangeInfo
- Wait all segments in OnlineList to be loaded
- Add handoff event into pending list
- Wait all segments in OfflineList is not used (reference count = 0)

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-25 11:51:46 +08:00
Ten Thousand Leaves 4ef2df8cb9
Fix an issue where query coord calls MinioChunkManager during init (#16629) 
It causes Milvus not runnable with local storage.

issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:11:46 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-24 22:03:44 +08:00
congqixia bd29518b6e
Remove debug log with no meaning (#16583) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-22 13:11:42 +08:00
congqixia 6de1d223a6
Remove debug log in querynode segment/node detector (#16572) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-22 11:07:42 +08:00
congqixia 69252f812d
Implement memory replica in Proxy, QueryNode and QueryCoord (#16470) 
Related to #16298 #16291 #16154
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-20 16:15:41 +08:00
codeman 4e12271d79
Support login with username and password (#15656) (#16341) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-11 19:49:34 +08:00
Cai Yudong 71cd7ba67a
Add configuration common.indexSliceSize (#16438) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-04-08 20:29:33 +08:00
xige-16 27b4cbc098
Cherry pick remove translateHits commit to mater (#16436) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-08 20:27:31 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
congqixia 4cb7c28bfa
Fix ReplicaInfo package name (#16400) 
Fix #16399
PR 16373 breaks the build after ReplicaInfo moved from querypb to milvuspb
Also fix  problem in Makefile
Limit action/setup-go version to ~1.17.1(>=1.17.1 <1.18.0)

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-06 19:47:31 +08:00
congqixia 18d76261a0
Add ShardNodeDetector implementation (#16373) 
Use etcd as a ShardNodeDetector
Unit tests is based on embedded etcd

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-06 15:43:31 +08:00
bigsheeper 2b27fa4609
Print numTimestamps instead of Timestamps (#16369) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-06 14:47:31 +08:00
bigsheeper 4f4a417c8f
Unify in log print (#16368) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-06 14:09:35 +08:00
Letian Jiang 09b90fd8bf
Fix QueryCollection retrieve unit test (#16376) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-04 21:07:30 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
congqixia aa1c26de77
Add ShardCluster implementation (#16360) 
ShardCluster maintains shard replica meta information
It watches node & segment change events
Provides shard replica search/query services

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-02 17:35:29 +08:00
Letian Jiang 028c5cb882
Modify grpc interface for replica Search/Query in QueryNode (#16326) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-02 14:15:31 +08:00
congqixia 7a44fff8cd
Use fmt.Errorf instead of string concat in tsafe_replica (#16344) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-01 18:37:28 +08:00
Letian Jiang cb65860e22
Add implementation for QueryShardService (#16317) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-01 14:19:29 +08:00
Letian Jiang ad9a2217ae
Add interface for QueryShardService and QueryShard (#16309) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-31 14:05:28 +08:00
congqixia 899638810a
Add error log when tsafe register watcher more than once (#16294) 
Add some test case to cover the error case
Add log in error branch

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-30 22:07:28 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
Cai Yudong 5a9e1af3bb
Update create plan CGO interface (#16271) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-30 17:13:27 +08:00
congqixia 18a3e9f265
Add grpc interface for replica Search/Query in QueryNode (#16197) 
Resolves #16195
Add Search and Query grpc interface in query proto and types/types.go

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-30 12:03:27 +08:00
jaime d5d6e115bd
Refine delete segment data log (#16241) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-29 18:41:27 +08:00
congqixia 91d5436807
Improve log level and param function in QueryCollection (#16254) 
Use debug level for all debug information
Use specific parameter function for appliable ones

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-29 12:31:27 +08:00
bigsheeper 9d673006e0
Improve primary keys print (#16232) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-28 23:15:27 +08:00
xige-16 d2424fc50a
Print msgID instead of loadInfo when load segment (#16238) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-28 23:13:27 +08:00
godchen 478f6ca11e
Add cache for vector chunk (#15912) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-26 22:05:26 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
bigsheeper 7fee4743f5
Improve log print in OOM error info (#16170) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-24 20:17:25 +08:00
yah01 63a71e247a
Loading segments concurrency level fallbacks to the value that won't cause OOM (#16172) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-24 13:39:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
Jiquan Long f8d9bc919d
Unify interface of vector index & scalar index. (#15959) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-21 14:23:24 +08:00
yah01 31c5b286da
Fix wrong unittest, check whether loading collection will cause OOM (#16068) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-18 13:17:22 +08:00
godchen 78557ca6d7
Replace minio kv with minio chunk manager (#15936) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-17 18:03:23 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
Jiquan Long ffdad82b9c
Set load info in reschedule tasks (#16065) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-16 11:43:21 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
congqixia b61e8201e7
Refactor querynode historical replica (#15992) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-15 19:15:20 +08:00
jaime 97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-15 14:45:22 +08:00
bigsheeper caf9cbfcd4
Fix load failure and remove parition release related code (#16038) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-14 23:18:02 +08:00
congqixia 6a66faa2fb
Explictly specify Load operation meta (#16027) 
Fix #15996

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-14 09:50:01 +08:00
jaime 29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
yah01 4f3d5fb6eb
Make load operation parallel improve performance (#15928) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-11 14:28:01 +08:00
bigsheeper 9ad5c14a4a
Return error when create message stream failed in QueryNode (#15317) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-09 15:17:59 +08:00
顾炎 4f8c08f912
fix checkSegmentSize overflows int64 (#15858) 
Signed-off-by: guyan <632722781@qq.com>
 2022-03-09 13:29:58 +08:00
zhenshan.cao dff08dbf47
Refactor querynode unittest (#15929) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-08 17:39:58 +08:00
bigsheeper 6fec7ed6fb
Add search benchmark in QueryNode (#15854) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-08 10:15:59 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
cai.zhang 88527fc0a9
Fix metrics label conflict (#15839) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-03 16:05:57 +08:00
Cai Yudong 92c8e32ebd
Let MemoryKV.Load return error when key not exist (#15814) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-02 18:51:55 +08:00
bigsheeper 12eb8c3da5
Add prometheus metrics for QueryNode (#15649) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-02 14:49:55 +08:00
bigsheeper 6a48071a5c
Set BaseMsg for LoadSegmentRequest, prevent crashing in log printing (#15725) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-02-26 20:09:54 +08:00
godchen fcf0887d92
Add and implement chunk manager interface. (#15541) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-24 23:53:53 +08:00
zhenshan.cao 142848fcc3
Abandon using protobuf to pass binaryset parameter (#15626) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-02-18 18:39:50 +08:00
congqixia c7f55c2e34
Make SessionWatch keep watch even no Rewatch func when find ErrCompacted (#15497) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-15 15:07:48 +08:00
yah01 684110bc9a
Avoid to copy during converting C bytes to Go bytes (#15551) 
* Avoid to copy during converting C bytes to Go bytes

Signed-off-by: yah01 <yah2er0ne@outlook.com>

* Add BytesConverter to achieve no-copy converting C bytes to Go bytes

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-15 14:17:48 +08:00
Xiaofan a18d6b46a4
Fix watch dml channel fail because of no collection meta (#15436) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-10 13:19:46 +08:00
Cai Yudong 8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-09 14:41:45 +08:00
jaime 307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
xige-16 55cdc5aa35
Get indexInfo and segment size by queryCoord (#14207) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-02-08 21:57:46 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
Ji Bin d83f69bb21
Support windows compliation (#15448) 
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS

issue: #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-02-07 23:43:46 +08:00
congqixia d830791e4f
Respect SegmentIDs in querynode.GetSegmentInfo (#15455) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-07 15:35:44 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
zhenshan.cao 5bc6ec7250
Fixbug: QueryNode search exceptions will cause memory leak (#15305) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-24 21:19:39 +08:00
xige-16 1cd2363bd8
Delete usless params SkipQueryChannelRecovery (#15289) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-21 21:43:40 +08:00
xige-16 57383c9f76
Delete Useless timetick channel (#15296) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-21 14:31:39 +08:00
xige-16 c132302129
Clean flowGraph if watchChannel failed (#15303) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-20 10:01:38 +08:00
xige-16 bbb672cb00
Remove state service in querynode (#15281) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-18 18:43:37 +08:00
xige-16 dfc6670f0f
Unsubscribe channel after query node down (#15230) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-17 17:37:37 +08:00
Jiquan Long 675e6d352b
QueryNodes send search/query results by rpc (#15223) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-17 14:41:35 +08:00
Xiaofan f617c8887b
Time Tick Failed need log (#15225) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-01-14 23:55:34 +08:00
bigsheeper 0c98f21d4d
Remove global sealed segments in historical (#15177) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-12 17:01:38 +08:00
bigsheeper ebed1a68ff
Add log for segcore search (#15159) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-11 18:07:34 +08:00
Cai Yudong 951249c6c1
Use KnowhereConfig for indexnode and querynode (#15112) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-11 09:35:34 +08:00
xige-16 99315ce00e
Optimize recentlyModified's lock (#15120) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-10 23:17:36 +08:00
bigsheeper 018f32847f
[skip e2e] Add comment for createSearchPlan (#15111) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-10 22:36:06 +08:00
bigsheeper 4174daee80
[skip e2e] Add comment for newPartition (#15110) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-10 22:34:09 +08:00
bigsheeper 67e8c09df5
[skip e2e] Add comment for removeSegmentID (#15109) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-10 22:32:14 +08:00
Cai Yudong b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 17:29:34 +08:00
xige-16 97ea9786e1
[skip e2e]Improve getSegmentByIDPrivate log format (#15028) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-10 10:11:34 +08:00
xige-16 43b09c83d1
[skip e2e]Improve getPartitionByIDPrivate log format (#15027) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-10 10:09:39 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
xige-16 708dfddba6
[skip e2e]Improve getSegmentInfosByColID log format (#15026) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-07 17:55:23 +08:00
bigsheeper 8d24428c4e
[skip e2e] Add comment for addSegmentID (#15020) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-07 17:47:48 +08:00
bigsheeper a7ad3f2a76
[skip e2e] Add comment for getSystemInfoMetrics (#15019) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-07 17:45:56 +08:00
bigsheeper 36b9c498ab
[skip e2e] Add comment for appendIndex (#15018) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-07 17:44:04 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
xige-16 f808390f32
[skip e2e]Improve getCollectionID error log (#14964) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-06 23:43:22 +08:00
xige-16 9dbb37c7ea
[skip e2e]Improve addCollection error log (#14963) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-06 23:41:28 +08:00
bigsheeper f90d5395a1
[skip e2e] Add comment for appendFieldInfo (#14958) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-06 23:35:47 +08:00
bigsheeper d38ad2f511
[skip e2e] Add comment for appendIndexParam (#14957) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-06 23:33:55 +08:00
xige-16 f7368e35a4
Add log for LoadSegmentsTask (#14703) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-06 23:26:18 +08:00
bigsheeper c79cbd341a
[skip e2e] Add comment for deleteLoadIndexInfo (#14956) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-06 23:22:03 +08:00
xige-16 0004238669
[skip e2e]Improve AddQueryChannel log (#14962) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-06 23:17:26 +08:00
xige-16 851794598e
[skip e2e]Improve log format (#14886) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-05 21:17:21 +08:00
xige-16 b3d33a78f5
[skip e2e]Improve log format (#14887) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-05 21:15:20 +08:00
xige-16 d751357be1
[skip e2e]Improve log format (#14885) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-05 21:13:20 +08:00
bigsheeper 64a5db66e9
[skip e2e] Add comment for newLoadIndexInfo (#14883) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-05 21:05:20 +08:00
bigsheeper fe730024ab
[skip e2e] Add comment for newIndexLoader (#14882) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-05 21:03:26 +08:00
bigsheeper 8b9ce5780d
[skip e2e] Add comment for setIndexInfo (#14881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-05 20:59:30 +08:00
bigsheeper 12f37ffb57
Improve timeout log when query in QueryNode (#14766) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-05 10:35:31 +08:00
xige-16 abf1f58abe
Add log for watchDeltaChannelTask's nodeID (#14702) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-04 20:45:24 +08:00
godchen 9e51591b3a
Fix pulsar reader not close (#14543) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-04 14:45:19 +08:00
bigsheeper bd7a1cd3d1
[skip e2e] Add comment for getIndexInfo (#14745) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-04 14:06:02 +08:00
bigsheeper cda172ede2
[skip e2e] Add comment for estimateIndexBinlogSize (#14744) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-04 14:04:03 +08:00
bigsheeper 6bace04dd3
[skip e2e] Add comment for getIndexBinlog (#14743) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-04 14:02:12 +08:00
bigsheeper 2e6fe7d815
[skip e2e] Add comment for printIndexParams (#14694) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-31 19:09:19 +08:00
bigsheeper c95deb2266
[skip e2e] Add comment for loadIndex (#14693) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-31 19:07:23 +08:00
bigsheeper 6b806fc850
[skip e2e] Add comment for getReadyLoad (#14692) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-31 19:05:24 +08:00
xige-16 148dad23ac
Format watchDmChannel's log (#14633) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-12-30 20:27:20 +08:00
bigsheeper 8a3a1b556a
Free OS memory in QueryNode and IndexNode (#14305) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-30 20:03:29 +08:00
bigsheeper 983d9b4b73
[skip e2e] Add comment for getIndexParams (#14580) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-30 16:57:33 +08:00
bigsheeper 26a8bac54c
[skip e2e] Add comment for getIndexPaths (#14579) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-30 16:56:04 +08:00
bigsheeper 168f05de3a
[skip e2e] Add comment for getFieldID (#14578) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-30 16:39:04 +08:00
Jiquan Long ebb9b24b47
Improve name of flowgraph node (#14538) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-30 10:33:46 +08:00
xige-16 8b919f50c1
[skip e2e] Add debug log for watchDmChannelReq (#14551) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-12-29 21:51:45 +08:00
godchen eb7dcf9ed3
Fix do unsolved msg goroutine not release (#14536) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-29 21:06:03 +08:00
bigsheeper 20178360ba
[skip e2e] Add comment for getBuildID (#14520) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-12-29 17:39:04 +08:00