c468481a36
Optimize MqTtMsgStream and unittest ( #5498 )
...
* code optimize
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add getTsMsgFromConsumerMsg
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename some variables for better readability
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* code optimize
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* code optimize
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add chanWaitGroup
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add chanTtMsgTime
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename lastTimeStamp to currTimeStamp
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add comments
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* code optimize
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* support unsynced ttMsg
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* code optimize
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update testcase
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update testcase
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update testcase
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add TestStream_PulsarTtMsgStream_3
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix unittest
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* remove debug log
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* change ttMsg logic back
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix ci block issue
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update testcase
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix codacy
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-31 11:31:30 +08:00
ca0993c274
Update OWNERS
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-30 18:58:47 +08:00
b414800d49
Update Seek interface ( #5492 )
...
* update Seek
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update Seek for mqTtMsgStream
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* use Retry in Seek
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix static-check
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-29 23:21:34 +08:00
fce792b8bf
Add historical and streaming module in querynode ( #5469 )
...
* add historical and streaming
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* fix GetSegmentInfo
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* pass regression test
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2021-05-28 10:26:30 +08:00
3630eec92c
Fix receving wrong msg after seek ( #5441 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-05-27 02:38:37 +00:00
6766169878
Refactor repack logic for insertion ( #5399 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-05-25 11:53:15 +00:00
3e73775cc1
Add ComputeProduceChannelIndexes for msgstream ( #5364 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-05-24 08:30:09 +00:00
f18dfb4ff6
Add RetrieveTask implementation ( #5313 )
...
Resolves : #5257
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-05-20 07:02:31 +00:00
9151e13480
Remove not using dependence and proto ( #5226 )
...
- FlushMsg proto message is no longer in use
- We don't depend on "github.com/prometheus/common/log"
Signed-off-by: yangxuan xuan.yang@zilliz.com
2021-05-14 08:50:55 +00:00
e5d4963ba0
Distinguish the dsl and expr in proxy and query node ( #5118 )
...
Pass a parameter `IsExpr` from proxy to query node, so query
node can create search plan according to the variable `IsExpr`.
It makes `segcore` unnecessary to distinguish `dsl` or `expr`
using dynamic exception. When `IsExpr` is set to true, query
node will call `CreatePlanByExpr` according to the passed
information about expression. Otherwise query node will keep
still to use `CreatePlan` according to the `dsl` information. At
the same time, this pr adds some unittests to `CreatePlanByExpr`,
these unittests translate already exist case with `dsl` to case with
`expr`.
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-05-07 07:20:47 +00:00
82ccd4cec0
Rename module ( #4988 )
...
* Rename module
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
51b2b45e03
Add copyright for msgStream
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-04-19 11:30:19 +08:00
78155d3959
Refactor load Release to async call in query service
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-04-15 15:15:46 +08:00
952f70af2a
Clean makefile
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2021-04-09 14:07:03 +08:00
57831b9978
Add unittest to insertChannelsMap in proxy
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-04-08 15:41:28 +08:00
bfb7ca0df6
Add unittest to time tick logic in proxy service
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-04-07 10:06:17 +08:00
87a1e0b662
Reorganize msgstream
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-02 13:48:25 +08:00
8b62790017
Reorganize mqclient
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-02 10:01:11 +08:00
cdc96d1c32
Add unittest for msgstream_impl
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-04-01 17:29:21 +08:00
604f96ed2b
Replace consumers map by sync.map
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-31 10:05:24 +08:00
e823b56040
Add retry for seek function
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-30 13:53:49 +08:00
72917467ec
Change singlenode logs output
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-03-30 10:52:42 +08:00
c4f9a6f22b
Fix get position wrong
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-27 17:39:12 +08:00
35656ea9ce
Update pulsar client version and fix cannot close tcp connection error
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-27 13:58:52 +08:00
3ab8b9af25
Dlete unlocked mutex in seek function
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-27 11:16:20 +08:00
a48d46cfaf
Alter msgID’s type to bytes
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-27 09:46:54 +08:00
432b53149b
Refactor msgstream
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-26 20:10:11 +08:00
85bc2a54f0
Fix load twice hang bug
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-25 18:49:41 +08:00
7ad9b36207
Update reviewdog/action-hadolint github action version to v1.16.1
...
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-25 14:41:46 +08:00
c2734fa55f
Fix bug and enchance system
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-22 16:36:10 +08:00
2280791128
Add memory message stream for search
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-03-19 20:16:04 +08:00
0823382876
Refactor rmq_msgstream
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-19 19:33:21 +08:00
207f6ae773
Fix rocksmq CreateConsumerGroup
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-16 21:37:25 +08:00
32660d547b
Add endPositions in datanode
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-16 17:55:42 +08:00
fc6472eb08
Implement RocksMQ client
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-03-13 11:46:50 +08:00
f3649f0419
Refactor interface and proto
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
09ae985daa
Fix wrong error code in master_service_test.go
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-10 22:06:22 +08:00
7b2d67242d
Fix describe segment if index not exist
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-10 14:45:35 +08:00
0d499a9f99
Fix search waiting for seconds in singlenode
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-09 17:42:13 +08:00
455cc59dbb
Update publish-test-images.yaml
...
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-09 13:55:35 +08:00
33d8e13f96
Fix load-collection in singlenode
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-05 18:14:59 +08:00
ddddd65d10
Delete internal/errors package
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-05 10:15:27 +08:00
36bae90635
Fix singlenode search timeout caused by selectcase in rmq_msgstream
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-04 15:53:17 +08:00
6cdd4d02fc
Refactor seek function
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-04 14:49:51 +08:00
795cb74c85
Fix message stream miss ack
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-02 13:15:49 +08:00
f5977a1302
Change retry times and add retry logic
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-26 15:17:47 +08:00
84110d2684
Add tests/benchmark and tests/python_test using new python SDK
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2021-02-25 17:35:36 +08:00
7cb28428ce
Prevent consuming after loading collection
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-25 15:08:50 +08:00
a391d4fbbf
Fix stuck problem of create_index
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-25 12:03:35 +08:00
0679954922
Refactor tso and global id allocator
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-24 17:12:06 +08:00
b02a9d6b43
Enlarge rocksmq stream buffer size
...
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-02-24 15:16:09 +08:00
fd562f9f9c
Update doc: add proxy graph
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-24 09:48:17 +08:00
7a7a73e89c
Fix high memory usage in pulsarTtStream
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-23 11:40:12 +08:00
ea1d9ea99a
Replace log in dataservice
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-23 09:58:06 +08:00
e786ad6aa8
Fix inconsistency of produce msg and consume msg with rmq
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-22 16:34:15 +08:00
b07a371f25
Create a goroutine to receive msg for every consumer
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-22 09:58:34 +08:00
7ac1821cf1
Add RocksMQ throughout test
...
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-02-20 18:30:37 +08:00
346b0a9549
Add unittest for RmqTtMsgStream
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-20 15:58:19 +08:00
5769c000a1
Find failed python tests and add skip mark "r0.3-test"
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-19 15:37:04 +08:00
d8f41db823
Fix search crash error
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-19 13:56:01 +08:00
5b61182140
Add zap log
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-18 19:53:55 +08:00
af3736aff6
Fix rocksmq bug
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-18 14:13:55 +08:00
70fe5233cf
Watch queryChannel on query node
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-09 17:09:26 +08:00
833f1d49ab
Let singlenode use rocksmq
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-09 15:57:10 +08:00
dc3736281a
Integrate message stream
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-08 14:30:54 +08:00
2ca53fa668
Fix msgstream deadlock when loadCollection
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-07 15:47:10 +08:00
563a8d66ae
Add RmqTtMsgStream
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-07 15:10:13 +08:00
099220394e
Add rmq_msgstream unittest and fix bugs
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-06 11:39:12 +08:00
dd72dfcbd1
Fix queryservice start
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-05 17:57:41 +08:00
c231f6da79
Fix collectionName not found error
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-05 16:32:03 +08:00
57ba236117
Add mutex for message stream’s asConsumer
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-05 15:42:53 +08:00
c2ca2c276f
Fix response check error
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-04 19:34:35 +08:00
e3fadc45d4
Fix for new msgstream interface
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-04 15:23:21 +08:00
b89e5a3240
Add GetQuerySegmentInfo
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-04 14:37:12 +08:00
71b6c88c6e
Defer lock's unlock in meta::DropSegment
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-04 11:52:10 +08:00
8d35e76ad1
Change Msgstream interface
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-02-04 13:55:47 +08:00
6e7e0b748a
Send timetick to insert channel
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-03 20:04:29 +08:00
fd9e42d3ed
Update msgstream/unmarshal interface
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-02-03 17:30:10 +08:00
abffdbff62
Remove partition tag
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-03 16:44:38 +08:00
35f6f4b7c2
Add consume in rmq_msgstream
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-03 15:18:54 +08:00
4cd315408b
Refactor query service’s meta interface
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-02 11:52:41 +08:00
c2914dd113
Refactor segment allocator, use collectionID and partitionID
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-31 10:15:54 +08:00
854accf95b
Remove master and writenode
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-30 16:41:47 +08:00
f940cc455a
Add dockerfile for index
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-26 09:38:40 +08:00
a8fd6e7686
Enable drop index/field_data and extract info visitor
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-25 18:59:44 +08:00
5600d06583
Add global_rmq
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-25 18:53:08 +08:00
5aec8bc5e6
Refactor masterservice
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-24 20:26:35 +08:00
92e3c519d9
Use FlushCompletedMsg instead
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-23 10:40:34 +08:00
e68d50f29e
Invalid collection meta caceh && ingore time stamp if not dd request
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-23 10:12:41 +08:00
50e2369000
Change name and tag to ID
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 19:36:09 +08:00
377ee8687d
Modify flush logic
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 09:36:40 +08:00
ec27788628
Add segment statistics updates logic for datanode
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-21 09:55:25 +08:00
76bc365181
Add seek function for pulsarTtMsgStream
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-20 17:34:50 +08:00
4588342fa1
Add IdAlocator service
...
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-01-20 15:02:23 +08:00
7ee8623e1a
Refactor msgstream
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-01-20 10:02:59 +08:00
2ec573d2f9
Recover delete docs
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-20 09:36:50 +08:00
fb493cc235
Update interface name of GetIndexStates
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-19 18:32:57 +08:00
918e279869
Set msg stream
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-19 16:14:13 +08:00
76d92e73d1
Replace old proto with new proto
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-18 21:39:56 +08:00
d22c2cd203
Add proto for query service loading collection
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-18 19:32:08 +08:00
Cai Yudong
Xiangyu Wang
xige-16
zhenshan.cao
yukun
XuanYang-cn
dragondriver
quicksilver
groot
godchen
neza2017
sunby
cai.zhang
bigsheeper
BossZou
FluorineDog