milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,506 Commits
133 Branches
115 Tags
559 MiB
Commit Graph

124 Commits (9c9ab08f5499df77fe9c219e341f1315ed46f6b5)

Author SHA1 Message Date
Enwei Jiao 264f08953d
Refactor Rocksmq's config (#21304) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-26 19:11:30 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-16 15:59:23 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-05 20:55:17 +08:00
smellthemoon a46a5ac900
Fix rocksmq incompatible (#20888) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-30 10:01:15 +08:00
smellthemoon b5f178f22b
Fix kafka panic (#20860) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-28 18:17:14 +08:00
smellthemoon d9185485f1
Remove TraceID in mq (#20697) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-25 23:11:12 +08:00
jaime c761de19bc
Fix broken kafka UT (#20743) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 20:43:13 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 18:59:09 +08:00
wei liu 701e7b5a6e
support pulsar tenant and auth (#19727) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-11-11 10:55:05 +08:00
jaime e5419b3bc7
Remove kafka consumer ack (#20424) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-09 14:27:03 +08:00
SimFG deb9963d0e
Delete unused messages for the mq (#20295) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-03 21:41:35 +08:00
congqixia e1e3ed486e
Fix rmq consumer cancel may block forever (#20269) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-02 19:01:35 +08:00
jaime 48f5c60070
Refine msgstream close (#20194) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-31 15:13:33 +08:00
aoiasd e470cd3dfa
Move Rocksmq page info log to Rokcsmq monitor (#20022) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-10-27 15:13:31 +08:00
Xiaofan 1f170ce1ba
Refine kafka consumer (#19846) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 13:23:30 +08:00
smellthemoon 581e1d8c35
Fill MsgBase ALL (#19912) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-10-21 15:57:28 +08:00
aoiasd 7fe8e50689
print some group info in rocksmq monitor (#19863) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-10-20 16:05:27 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 17:51:24 +08:00
aoiasd 0a77561e78
Fix some message in rocksmq could not get acked timestamp (#19592) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-10-11 23:17:23 +08:00
Xiaofan 6826964e60
Rename some interface and files in id allocator (#19605) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-09 10:06:58 +08:00
bigsheeper d6406387f8
Dedup only for dml messages in msgstream (#19591) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-09-30 17:52:55 +08:00
jaime 8ac30397dd
Disable kafka msgstream ut (#19260) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-19 16:24:51 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
Xiaofan eabdc2b114
Disable pulsar batch and change background flush goroutine to larger interval (#18888) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-09 11:26:35 +08:00
Jiquan Long a5e2d6b6fb
Refactor RootCoord (#18930) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
 2022-09-05 13:29:11 +08:00
Xiaofan 10d1140885
Fix RocksMQ flaky test (#18954) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-01 19:46:59 +08:00
Xiaofan 8e22d03cf3
Support trigger compaction everyday to eliminiate deleted data in rocksmq (#18752) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-08-29 10:56:55 +08:00
wgcn d443b5420a
add user special kafka config (#18742) 
Signed-off-by: wgcn <1026688210@qq.com>

Signed-off-by: wgcn <1026688210@qq.com>
Co-authored-by: wanggang11335 <wanggang11335@autohome.com.cn>
 2022-08-25 11:02:53 +08:00
jaime 909e46b6c2
Fix LessOrEqualThan method of rocksmq (#18789) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-23 16:56:52 +08:00
wei liu 5c4f854271
fix load segment hangs (#18770) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-08-23 16:52:52 +08:00
jaime c345df41a7
Fix kafka ut (#18527) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-04 16:02:35 +08:00
jaime 4a32c1bb8e
Enable kafka ut with a mocked kafka server (#18366) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-03 19:06:35 +08:00
jaime 6e7831470c
Revert settings of Pulsar address and refine config code (#18494) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-02 21:26:33 +08:00
jaime 3ec147acfe
Upgrade confluent kafka go version to 1.9.1 (#18293) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-22 13:28:28 +08:00
congqixia d90308e9a0
Handle topic not exist in rqm & mqtt seek (#18314) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-20 15:42:29 +08:00
bigsheeper 1da970af18
Remove useless statsService in QueryNode (#17395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-11 15:12:25 +08:00
Xiaofan 653dce4948
Fix pulsar error check (#18207) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-11 10:56:25 +08:00
congqixia e61f61a9bd
Dedup insert msgs by MsgID in mqtt MsgPack (#18180) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-08 20:42:22 +08:00
Jiquan Long 6df47c7faf
Change pulsar compression type to zstd-faster (#18121) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-07 11:16:22 +08:00
Jiquan Long 44024c1b1d
Replace pulsar compression type zstd with lz4 (#18085) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-05 22:06:21 +08:00
Xiaofan b31f937301
Fix unsubscribe with subscription not exist (#18042) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-05 10:12:20 +08:00
Xiaofan a803f9e4b0
Fix Wait Group Race in Msgstream (#17931) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-30 10:38:18 +08:00
Bingyi Sun 24b3ad22d7
Fix msgstream close block. (#17880) 
issue: #17513
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-29 16:24:20 +08:00
jaime 762e3b78a6
support configurable sasl mechanisms for kafka (#17542) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-29 10:32:17 +08:00
Xiaofan e36fce1fd4
Add log for mqtt msgstream seek (#17640) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-20 15:18:13 +08:00
Xiaofan d70f483806
Support rocksmq dynamic lru cache size (#17626) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-20 10:56:12 +08:00
Xiaofan 1f6fbf91b2
Fix pulsar unsubsribe issue (#17562) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-16 17:28:11 +08:00
jaime dbef4c6ec2
Align kafka consume from latest with puslar and rocksmq (#17433) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-08 19:00:07 +08:00
xige-16 8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel (#17308) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-02 16:56:04 +08:00
jaime 5698bf4236
Add sasl configurations for kafka (#17323) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-02 12:12:03 +08:00
bigsheeper e0cbacba59
Check ignorable error to prevent unnecessary panic (#17317) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 10:12:03 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
Xiaofan 65269a4744
Remove rocksmq reader (#17149) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-23 13:55:59 +08:00
jaime a20e0dfc74
Refactor kafka msgstream implementation (#17069) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-05-23 12:39:58 +08:00
jaime 328ab7ecf6
Fix kafka client hang after killing broker pod of parition leader (#16983) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-05-14 23:05:54 +08:00
Xiaofan 000c5ff3de
Fix msgstream unsubscription (#16883) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-10 19:43:52 +08:00
Xiaofan 62658dcda6
Fix pulsar unscubsribe fail because of consumer not found (#16839) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-09 12:07:52 +08:00
Eng Zer Jun 5375161777
test: use `T.TempDir` to create temporary test directory (#16758) 
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Fixes: #16759
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-05-04 11:35:50 +08:00
jaime bb9ccbb7e2
Use a singleton kafka producer (#16739) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-29 17:29:47 +08:00
yah01 2022a361c0
Enable ZSTD compression for pulsar (#16014) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-24 10:25:43 +08:00
jaime c7e98b922a
Fix mq factory ut (#16483) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-14 10:45:34 +08:00
jaime ed3c29eae7
Support Kafka (#16149) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-12 19:47:33 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
Bingyi Sun 256ccb8f65
Fix rootcoord goroutine leak (#16071) 
issue: #15658
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-03-16 16:47:21 +08:00
Bingyi Sun d3d5a41b76
Fix memory leak casued by unclosed channel in msgstream (#16051) 
issue: #16045
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-03-15 15:55:21 +08:00
jaime 97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-15 14:45:22 +08:00
jaime 29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
XuanYang-cn c2fe12bc97
Handle error in topic creation of rocksmq (#15922) 
Reopen for: #14429

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-09 15:47:59 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00