Commit Graph

1478 Commits (50672f0d609317898448888748678a2063a9978a)

Author SHA1 Message Date
godchen c2ecce61c0
Make pulsar client singleton (#5782)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 19:03:56 +08:00
neza2017 96cb24e566
fix session util (#5780)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 18:53:55 +08:00
zhenshan.cao 5c69c6902d Refactor proxy channelTimeTick and fix search timeout (#5760)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:36 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
zhenshan.cao f74ea3beee [skip ci] Remove ProxyService (#5738)
* [skip ci] Remove ProxyService

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Change helm branch

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: paramstable crashed

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:35 +08:00
neza2017 ac9dde7352 fix seek on pulsar msgstream (#5726)
* fix seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* test seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* msg-seek test test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add pure pulsar client seek test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:14:29 +08:00
Cai Yudong fd07d544c3 [skip ci]add timerecord for debug (#5729)
* add timerecord for debug

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update comments

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:14:29 +08:00
sunby 189ac881f3 Fix bugs (#5676)
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
godchen 91ef35bad4 Change receive msg logic (#5605)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
neza2017 cb6ae3678a fix master service ut (#5555)
* fix master unitttest

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* tt

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby f1ccbb8f9a Fix init session in dataservice (#5522)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
bigsheeper 8aae0f7cc9
Refactor flow graph and load/watchDML in query node (#5682)
* Refactor flow graph and load/watchDML in query node

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-09 11:37:55 +08:00
Cai Yudong d5cd561449
Add metrics for master (#5515)
* metric

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* metric

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add MasterDDChannelTimeTick for metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add metrics for all master grpc

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* register metrics

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add metrics register

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add comments for codacy check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-01 11:04:31 +08:00
Xiangyu Wang ca0993c274 Update OWNERS
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-30 18:58:47 +08:00
yukun c63524deb9
Add dablooms (#5440)
* Add dablooms cwrapper

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix undefined reference bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add cwrapper_dablooms_build shell script

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-05-28 10:23:30 +08:00
neza2017 f80dbab6ec
watch proxynode (#5424)
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
2021-05-26 12:14:30 +00:00
yukun a3fb1356de
Fix rocksmq seek error (#5432)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>

In milvus-standalone, when seek is called, rocksmq won't put a
mutex in MsgMutex channel, which cause that the consume
process hang. So the seek function in rocksmq needs to 
notify the consumer to consume.
2021-05-26 09:31:09 +00:00
godchen 200801271d
Fix GetSession error (#5401)
Fix GetSession error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-25 12:15:46 +00:00
godchen 042a061123
Fix session util error (#5370)
Fix session util error.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-24 08:31:06 +00:00
godchen dd736ee8ba
Change session doc (#5355)
Change session doc

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-24 03:53:07 +00:00
zhenshan.cao 4e1b12269b
Fix bug: rand.Seed reset and generate duplicate number (#5361)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-05-24 02:50:37 +00:00
neza2017 e326eaad1e
master client,reconnect if grpc failed (#5360)
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-05-24 01:44:49 +00:00
godchen d189bf4d33
Change pulsar version (#5352)
Change pulsar version.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-22 03:36:49 +00:00
godchen 0eee7e5252
Change session (#5329)
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-21 11:28:52 +00:00
godchen 913712fbea
Improve session manager (#5310)
Improve session manager.
Issue #5174

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-20 07:07:25 +00:00
godchen e224d1e725
Add SessionManager (#5288)
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-19 10:36:05 +00:00
godchen ccc0130bb4
Add exclusive parameter (#5278)
Add exclusive parameter.
Issue #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-18 11:07:27 +00:00
godchen cca75583a6
Add session util (#5269)
Add session util.
Issue #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-18 03:39:21 +00:00
XuanYang-cn 9151e13480
Remove not using dependence and proto (#5226)
- FlushMsg proto message is no longer in use

- We don't depend on "github.com/prometheus/common/log"

Signed-off-by: yangxuan xuan.yang@zilliz.com
2021-05-14 08:50:55 +00:00
Cai Yudong 6f3a460b4e
Add pulsar seek example in pulsar_client_test.go (#5154)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-10 02:32:10 +00:00
FluorineDog 88f5642603
Add plan proto and support basic boolean expr parser (#5088)
**What type of PR is this?**
- [x] Feature

**What this PR does / why we need it:**
This PR supports boolean expression as DSL.
1. The goal of this PR is to support predicates
    like `A > 3 && not B < 5 or C in [1, 2, 3]`. 
2. Defines `plan.proto`, as Intermediate Representation (IR) 
    used between go and cpp. 
3. Support expr parser, convert predicate expr to IR
    in proxynode, while doing static check there
4. Support IR to AST in cpp, enable the execution
2021-04-29 08:48:06 +00:00
XuanYang-cn cebdfb34f7
Enable multiple datanode (#5068)
Now it's able to start multiple datanode, however, they will process the same insert data.
Further changes are needed to truly enable multiple datanode:

While registering, dataservice need to return different insert channels for different datanode.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-04-28 18:43:48 +08:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988)
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
sunby fad94fb576
Remove unsed configs (#4990)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-22 03:04:59 +00:00
Xiangyu Wang c731e2df78 Add license header in util
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-19 13:42:47 +08:00
XuanYang-cn e281f6a4a1 Remove useless configs in config files
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-04-13 10:04:39 +08:00
godchen deba964590 Change name of kvBase
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-04-12 18:09:28 +08:00
Cai Yudong 687d48b75e Improve code readibility
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2021-04-12 14:11:06 +08:00
yukun 964e7e0a28 Add unittest for mqclient
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-04-08 20:05:33 +08:00
dragondriver decc80a525 Add unittest to proxy service
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-04-06 14:12:57 +08:00
Xiangyu Wang 87a1e0b662 Reorganize msgstream
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-02 13:48:25 +08:00
Xiangyu Wang 8b62790017 Reorganize mqclient
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-02 10:01:11 +08:00
dragondriver 6796c7bfe7 Add unittest to function util
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-03-31 18:22:17 +08:00
yukun 18e0581d5c Destroy readOptions after use
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-31 14:10:07 +08:00
yukun 604f96ed2b Replace consumers map by sync.map
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-31 10:05:24 +08:00
sunby f8e879b423 Fix high cpu usage in singlenode
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-29 17:48:15 +08:00
sunby ff8ebd5f42 Fix bug of estimating record size
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-29 15:14:51 +08:00
yukun c4f9a6f22b Fix get position wrong
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-27 17:39:12 +08:00
xige-16 a48d46cfaf Alter msgID’s type to bytes
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-27 09:46:54 +08:00
yukun 432b53149b Refactor msgstream
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-26 20:10:11 +08:00
xige-16 7e60460aee close timeTickStream in flow graph when the collection is released
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-26 18:40:04 +08:00
neza2017 eca1e25470 Add msg id at master service
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-26 15:13:33 +08:00
dragondriver 41e1975611 Parallelize the processing of loading binlog and saving index files
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-03-26 11:19:02 +08:00
yukun 85bc2a54f0 Fix load twice hang bug
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-25 18:49:41 +08:00
quicksilver 7ad9b36207 Update reviewdog/action-hadolint github action version to v1.16.1
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-25 14:41:46 +08:00
bigsheeper 1634d75980 Release flowgraph properly
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-03-22 12:49:50 -05:00
紫晴 3be6dde459 Add smoke tag
Signed-off-by: 紫晴 <ting.wang@zilliz.com>
2021-03-23 14:35:30 +08:00
zhenshan.cao c2734fa55f Fix bug and enchance system
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-22 16:36:10 +08:00
Xiangyu Wang 9ba9bf5e8a Release memory after rocksdb.write()
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-03-19 20:20:32 +08:00
yukun 0823382876 Refactor rmq_msgstream
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-19 19:33:21 +08:00
yukun 207f6ae773 Fix rocksmq CreateConsumerGroup
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-16 21:37:25 +08:00
XuanYang-cn 32660d547b Add endPositions in datanode
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-16 17:55:42 +08:00
Xiangyu Wang fc6472eb08 Implement RocksMQ client
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-03-13 11:46:50 +08:00
godchen f3649f0419 Refactor interface and proto
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
xige-16 09ae985daa Fix wrong error code in master_service_test.go
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-10 22:06:22 +08:00
del-zhenwu b1763c3690 Disable xdist in test cases
Signed-off-by: del-zhenwu <zhenxiang.li@zilliz.com>
2021-03-10 15:27:26 +08:00
neza2017 7b2d67242d Fix describe segment if index not exist
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-10 14:45:35 +08:00
Xiangyu Wang c875fdf913 Add logs in rocksmq
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-03-10 10:02:55 +08:00
yukun 0d499a9f99 Fix search waiting for seconds in singlenode
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-03-09 17:42:13 +08:00
quicksilver 455cc59dbb Update publish-test-images.yaml
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-09 13:55:35 +08:00
ThreadDao 7a3223dc81 tags collection stats
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2021-03-08 10:09:48 +08:00
quicksilver 74154a11a4 Fix deploy error during regression stage
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-05 16:52:45 +08:00
sunby ddddd65d10 Delete internal/errors package
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-05 10:15:27 +08:00
godchen 3ba0dc2f44 Change doc
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-04 10:35:28 +08:00
bigsheeper 456d0e8c23 Refactor time tick align strategy
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-27 18:33:29 +08:00
BossZou 7893ebdb40 Fix test insert case
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-02-26 17:59:13 +08:00
godchen 7b9fdd7f29 Add opentracing
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-26 17:44:24 +08:00
Xiangyu Wang 38266b8569 Lock channel at produce and consume
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-02-25 17:44:29 +08:00
Cai Yudong 84110d2684 Add tests/benchmark and tests/python_test using new python SDK
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2021-02-25 17:35:36 +08:00
bigsheeper 7cb28428ce Prevent consuming after loading collection
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-25 15:08:50 +08:00
cai.zhang 1d899513dd Unify service name in config file
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-24 18:46:19 +08:00
zhenshan.cao 0679954922 Refactor tso and global id allocator
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-24 17:12:06 +08:00
zhenshan.cao 7c5e2d89f6 Read node port from config
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-23 18:08:17 +08:00
dragondriver b506754364 Add workflow of drop index in index service
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-23 11:57:18 +08:00
zhenshan.cao bbfcbbdd68 Fix ci: paramstable and startup logic refactor
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-23 11:40:30 +08:00
sunby ea1d9ea99a Replace log in dataservice
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-23 09:58:06 +08:00
yukun e786ad6aa8 Fix inconsistency of produce msg and consume msg with rmq
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-22 16:34:15 +08:00
sunby b07a371f25 Create a goroutine to receive msg for every consumer
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-22 09:58:34 +08:00
BossZou 7ac1821cf1 Add RocksMQ throughout test
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-02-20 18:30:37 +08:00
yukun 346b0a9549 Add unittest for RmqTtMsgStream
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-20 15:58:19 +08:00
groot af3736aff6 Fix rocksmq bug
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-18 14:13:55 +08:00
groot 833f1d49ab Let singlenode use rocksmq
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-09 15:57:10 +08:00
dragondriver da94fbed0a Parse to map when the key of index params is named params
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-07 21:32:37 +08:00
yukun 563a8d66ae Add RmqTtMsgStream
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-07 15:10:13 +08:00
yukun 099220394e Add rmq_msgstream unittest and fix bugs
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-06 11:39:12 +08:00
godchen c2ca2c276f Fix response check error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-04 19:34:35 +08:00
bigsheeper c6c99ef32b Implement getSegmengInfo function
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-04 11:40:14 +08:00
Xiangyu Wang fd9e42d3ed Update msgstream/unmarshal interface
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-02-03 17:30:10 +08:00
bigsheeper abffdbff62 Remove partition tag
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-03 16:44:38 +08:00
yukun 35f6f4b7c2 Add consume in rmq_msgstream
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-03 15:18:54 +08:00
xige-16 4cd315408b Refactor query service’s meta interface
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-02 11:52:41 +08:00
Xiangyu Wang 99cef4b5c8 Add Distributed main entrypoint
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-01-29 17:08:31 +08:00
zhenshan.cao ee0faddb07 Refactor proxy
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-29 09:27:26 +08:00
bigsheeper 4f914a2c30 Add docker file and main function of query node
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-27 09:50:52 +08:00
neza2017 d1b23cb8c5 Add debug info
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-26 19:24:09 +08:00
sunby f632ed0272 Fix data service init
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-26 18:45:12 +08:00
zhenshan.cao e83ac41bca Add component for index
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-26 18:01:32 +08:00
xige-16 7c210310dc Add load function for query service
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-26 15:13:20 +08:00
XuanYang-cn 6c303c67b2 Rearrange update segment statistics of start position procedure
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-26 09:44:39 +08:00
cai.zhang f940cc455a Add dockerfile for index
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-26 09:38:40 +08:00
FluorineDog a8fd6e7686 Enable drop index/field_data and extract info visitor
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-25 18:59:44 +08:00
yukun 5600d06583 Add global_rmq
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-25 18:53:08 +08:00
XuanYang-cn b794921b7e Add enum Role in internal.proto
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-23 18:56:08 +08:00
BossZou 01267776d2 Fix rocksmq consume invalid currendID
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-01-23 10:15:57 +08:00
xige-16 511aa419fa Refactor query service
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-22 14:28:06 +08:00
neza2017 9828a52999 Add create index
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-21 10:01:29 +08:00
FluorineDog 7f044fff82 Enable search for sealed
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-21 15:29:52 +08:00
sunby be6775671b Add error returned by Init,Start,Stop functions
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-19 17:38:01 +08:00
xige-16 76bc365181 Add seek function for pulsarTtMsgStream
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-20 17:34:50 +08:00
yukun 527c0c49df Add rocksmq unittest with goroutines
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-20 16:46:58 +08:00
BossZou 4588342fa1 Add IdAlocator service
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-01-20 15:02:23 +08:00
sunby 3602376a10 Rename GetServiceStates to GetComponentStates and reconstruct NodeStates and ServiceStates
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-20 11:02:29 +08:00
cai.zhang fb493cc235 Update interface name of GetIndexStates
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-19 18:32:57 +08:00
XuanYang-cn 6a13386393 Add DataNode package
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-19 11:37:16 +08:00
neza2017 1ce32d8775 Add master service
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-19 09:45:42 +08:00
yukun b874a55c35 Add complex unittest for rocksmq
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-19 10:22:27 +08:00
zhenshan.cao 76d92e73d1 Replace old proto with new proto
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-18 21:39:56 +08:00
yukun 3aca35b10c Add rocksmq unittest
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-18 15:05:49 +08:00
bigsheeper 48821690b3 Rename Service interface to Component, and update segmentLoad strategy
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-18 10:38:41 +08:00
XuanYang-cn 37688559f1 Fix multi rowdata writenode crash bug
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-16 15:06:19 +08:00
yukun 44c75bdd2a Add consume function in rocksmq
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-15 20:12:26 +08:00
BossZou 31178d6a5c Add procude logic of RocksMQ
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-01-15 16:06:35 +08:00
quicksilver 7fe061b846 Raneme init_devcontainer.sh to devcontainer.sh
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-01-15 14:38:36 +08:00
neza2017 56ab2bda2d Update proto:add partition id
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-15 11:53:22 +08:00
zhenshan.cao 63b21321d6 Refactor indexservice and update doc
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-15 13:50:27 +08:00
Xiangyu Wang 52bd6abdf8 Change RocksMQ interface
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-01-13 17:26:57 +08:00
GuoRentong f3d2afa4f4 Update doc:detailed interface definition
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-11 18:35:54 +08:00
quicksilver a6690dbc99 Fix syntax error: Bad fd number
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-01-11 15:25:19 +08:00
yukun 18cfd31d40 Fix Load and Remove with prefix
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-11 15:17:06 +08:00
sunby 5dfe9448ae Modify test_index
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-08 16:00:16 +08:00
GuoRentong 82ad488555 Update doc: rocksmq
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-08 18:37:24 +08:00
XuanYang-cn 65089ea362 Fix flush sync test bug and remove unecessary lock in meta_table
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-08 17:20:59 +08:00
GuoRentong c4599e26de Update doc: node api
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-08 16:59:19 +08:00
bigsheeper e13822043d Fix search test when partition not exists, add binary vector search validation
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-08 16:41:27 +08:00
godchen fa73f17f94 Add opentracing doc
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-01-08 15:32:38 +08:00
dragondriver ced56c84f8 Add support to build binary vector index
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-01-07 14:56:17 +08:00
godchen 8f1f118b26 Add query opentracing
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-01-07 14:24:35 +08:00
bigsheeper a1e2ff3c26 Add gcNode to write node
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-06 16:44:12 +08:00
neza2017 4015d7245d Merge operation
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-06 14:45:50 +08:00
bigsheeper 810be533ab Add gcNode, and remove msg's downstream index
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-06 14:02:25 +08:00
zhenshan.cao b06e01f523 Initmate querynode collection manager
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-06 11:31:36 +08:00
godchen 1c87faf8b3 Add insert request opentracing
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-01-06 11:17:35 +08:00
sunby 0a383608f3 Remove mock client in master
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2020-12-31 11:20:03 +08:00
FluorineDog 89f38e459a Fix bool type
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-04 17:52:42 +08:00
zhenshan.cao 6bd5702441 Add indexbuilder address to paramstable
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-04 14:53:20 +08:00
XuanYang-cn c0797333ce Add ddl binlog metaTable in writenode
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2020-12-24 20:55:40 +08:00
neza2017 4dbf4ade41 Add test for master
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-12-24 16:19:42 +08:00
bigsheeper 4ecdea698f Refactor query node and query serviceMain changes:1. Add ddBuffer and save binLog.
2. Trans to insertData.
3. Change dataFields data to Data, dim to Dim4. Add float vector and binary vector.5. Deserialize data and convert to InsertData.6. Move all data into InsertData.7. Add insert buffer and hash string.
8. Add minIOkV in insertBuffer node.
9. Init write node insertBuffer maxSize from writeNode.yaml.
10. Add ddBuffer.
11. Add ddBuffer binLog and minio.
12. Add ddNode unittest.
13. Remove redundant call.
14. Increase test time.
15. Delete ddl const, use request's timestamp instead.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-12-24 15:38:29 +08:00
neza2017 0afeaa78c8 Fix unittest of flow graph
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-12-24 14:10:00 +08:00
XuanYang-cn 7ce0f27ebc Add buffer to minIO for binlogs
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2020-12-23 18:06:04 +08:00
sunby 2aa483c922 Add index meta to meta table
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2020-12-22 20:16:32 +08:00
sunby c71cd40f68 Refactor segment manager
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2020-12-19 12:55:24 +08:00
bigsheeper 762b1e1f1f Add ddService, fieldID, requestID and refactor params table, fix nil hits search result bug
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-12-10 16:31:09 +08:00
XuanYang-cn 32977e270c Add impl cgo of parquet
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2020-12-08 14:41:04 +08:00
zhenshan.cao 7bbbc14637 Fix bug: address already used
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-12-07 15:22:20 +08:00
xige-16 37d7526d31 Fix search test failure
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2020-12-07 14:37:42 +08:00
neza2017 aeca8f85a5 Run cwrapper unittest in ci
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-12-05 17:39:58 +08:00
cai.zhang 806a97505c Add query node for CI progress
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-12-05 16:12:45 +08:00
neza2017 70710dee47 Add parquet payload
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-12-05 16:11:03 +08:00
FluorineDog 6412ebc0d4 Add support of metric type in schema, enable binary vector, fix segfault
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2020-12-05 06:46:01 +08:00
zhenshan.cao 0b6fe0a8cf Refactor paramtable
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-12-04 14:18:17 +08:00
godchen 7ab5b5d80d Add MinIO kv implements
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2020-12-03 19:00:11 +08:00
dragondriver 2358fae703 Check the status of search results before reduce in Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-30 19:08:32 +08:00
bigsheeper 0cd3e8d86c Use exception instead of runtime error
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-30 17:58:23 +08:00
cai.zhang 546beb333d Compile master and proxy without cgo
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-30 16:40:13 +08:00
quicksilver 86573d0053 Create Jenkins pipeline
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2020-11-30 10:07:04 +08:00
FluorineDog 77fa75b1ec Add binary insert and warper of binary search, rename vector
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2020-11-30 05:18:44 +08:00
cai.zhang 85a544b79b Add dockerfile for each component
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-28 19:06:48 +08:00
cai.zhang 115d0732af Update docker-compose file for unittest
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-28 16:43:39 +08:00
dragondriver eb6c0e0a28 Fix the bug when reduce the multiple search result from query nodes
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-28 15:57:49 +08:00
godchen 92bf09c7e6 Add ddstream and scheduler implements
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2020-11-28 15:26:39 +08:00
dragondriver 235e723eb6 Read proxy id from param table instead of hardcode
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-27 11:17:12 +08:00
XuanYang-cn 3e605ae25f Rename reader to querynode
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2020-11-27 10:56:53 +08:00
neza2017 32445dd02b Add cpu profile
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-26 16:01:31 +08:00
cai.zhang 57da66ba72 Update read config logic in docker
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-26 15:18:36 +08:00
bigsheeper 41fb2d2991 fix lock usage
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-24 16:12:39 +08:00
bigsheeper 8ec7e4576e Add dropPartition and createPartition
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-24 15:59:51 +08:00
neza2017 ad0078c07a Load grpc port of proxy from config
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-24 11:50:21 +08:00
cai.zhang 17c3a70d64 update ParameterTable
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-21 15:06:46 +08:00
neza2017 ce89f64bb3 Set master config
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-19 10:46:17 +08:00
neza2017 c3de667bf5 Implement segment management in master
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-18 19:46:18 +08:00
bigsheeper 99ea7e9f05 Update param table
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-18 19:33:56 +08:00
FluorineDog e7dd30a884 Add framework of ExecExprVisitor
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2020-11-18 17:32:52 +08:00
dragondriver b32e55d5f0 Add timeout logic to task in Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-17 20:00:23 +08:00
quicksilver 72ed65d7f8 Check builder images
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2020-11-17 18:36:03 +08:00
bigsheeper e117a6097a Fix pulsar url
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-17 16:32:34 +08:00
xige-16 4e8efe85df Delete wrong useage of *TsMsg
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2020-11-17 14:10:07 +08:00
cai.zhang 5bcabffdaf Replace conf struct with ParamsTable
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-16 21:10:43 +08:00
dragondriver aff1e79f82 Fix SearchTask in Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-16 17:01:10 +08:00
FluorineDog 8c48cf30c0 Format and and extra parameters for checking
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2020-11-16 15:41:56 +08:00
neza2017 9d3a21a9c9 Rewrite NewTSOKVBase
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-16 12:37:46 +08:00
dragondriver c2382adf04 Initialize the task scheduler of Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-14 11:24:49 +08:00
cai.zhang cfee642a11 Add ruleguard and open ruleguard check
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-13 10:00:32 +08:00
cai.zhang b68322d469 Open golangci-lint check
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-12 17:58:05 +08:00
cai.zhang 1cd2fd9954 Add the standard of golangci-lint
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-12 12:04:12 +08:00
godchen 4a11a6431b Add time sync producer
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2020-11-12 11:21:21 +08:00
bigsheeper c9c8fb9def Close pulsar channel
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-12 11:18:23 +08:00
cai.zhang 3b2c86c42d Fix bug for cmake_build dir was changed
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-12 10:45:46 +08:00
zhenshan.cao 057563d936 Refactor master
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-11 17:53:54 +08:00
bigsheeper b441685635 Refactor segment, partition and collection, refactor search and services
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-09 16:27:11 +08:00
zhenshan.cao 9c490a209b Delete unused code
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-05 19:11:12 +08:00
bigsheeper 88a3fa31fa Refactor query node and services
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-05 10:52:50 +08:00
zhenshan.cao e9122921b9 Use type alias
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-04 17:58:43 +08:00
neza2017 d18c17e786 Add Meta table implementation
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-04 16:28:14 +08:00
godchen bade037fb2 Add meta_table implementation
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2020-11-04 16:01:28 +08:00
bigsheeper d6fe379143 Add insertNode and serviceTimeNode
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-03 17:09:51 +08:00
zhenshan.cao da18bc4952 Update proto
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-03 16:38:16 +08:00
zhenshan.cao e962b57fbb Add timestamp allocator
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-03 14:53:36 +08:00
bigsheeper f49c98ed85 Add reader flowgraph about nodes and messages
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-02 19:30:12 +08:00
bigsheeper 8020dc2256 Add flow graph
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2020-11-02 16:44:54 +08:00
zhenshan.cao 27b9a51938 Add logic of allocate id
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-30 16:27:58 +08:00
GuoRentong f7bd6ebfad Add docs
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2020-10-29 20:42:47 +08:00
zhenshan.cao 6eee7d1342 Fix bug and update proto
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-29 19:55:57 +08:00
zhenshan.cao f0cc9c8ad3 Add tso for master and add timestamp
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-29 09:31:08 +08:00
zhenshan.cao 251bc2a19e Fix conf.LoadConfig failed to load yaml file
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-19 18:31:00 +08:00
GuoRentong 974496a2b2 Update issue templates
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2020-10-19 17:45:56 +08:00
zhenshan.cao 6a136c1a76 Change import path
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-19 17:39:51 +08:00
zhenshan.cao 64295db471 Refact master and proxy and add etcdutil
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-15 21:31:50 +08:00