8743752ac3
enhance: [2.4] force to stop buffer message when receiving the drop collection message ( #36917 )
...
/kind improvement
pr: #36916
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-17 12:53:29 +08:00
1955738ab8
enhance: [2.4] Produce messages of multiple topics in parallel ( #36344 ) ( #36462 )
...
Cherry-pick from master
pr: #36344
Related to #36343
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-08 18:31:20 +08:00
07e1bc8c08
enhance: [2.4] get msg type from the msg header to reduce the Unmarshal usage ( #36454 )
...
/kind improvement
- pr: #36409
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-26 16:41:15 +08:00
a35d99eabf
fix: [2.4] long buffering causes mq to be unable to receive messages. ( #36425 )
...
- issue: #36397
- pr: #36420
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:17 +08:00
fc324b4254
feat: [2.4] add the rbac msg and send them to the replicate channel ( #35562 )
...
- issue: #35391
- pr: #35392
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 14:45:00 +08:00
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 ( #34394 ) ( #35555 )
...
issue: #34252
pr: #34394 #35072 #35084
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
6423b6c718
enhance: move rocksmq from internal to pkg ( #34165 )
...
pr: https://github.com/milvus-io/milvus/pull/33881
issue: https://github.com/milvus-io/milvus/issues/33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 13:36:05 +08:00
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33793 )
...
issue: #33610
pr: #33792
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 14:14:04 +08:00
ff9bb17cc5
fix: Masked kafka credentials getting printed in logs ( #33747 )
...
Cherry-pick from master
pr: #33730
issue: https://github.com/milvus-io/milvus/issues/33727
Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-06-11 15:59:54 +08:00
c331aa4ad3
enhance: [2.4] add the includeCurrentMsg param for the Seek method ( #33743 )
...
/kind improvement
- issue: #33325
- pr: #33326
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-11 15:01:55 +08:00
d6bc95de55
enhance: [2.4] Add param item to ignore bad message id in checkpoint ( #33123 ) ( #33249 )
...
Cherry-pick from master
pr: #33123 #33158
See also #33122
This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-23 15:27:41 +08:00
b6fefee0cf
fix: etcd not connectable when auth enabled ( #31633 )
...
Fix etcd config source didn't respect auth enabled
Also removed pulsar recoverable error when pulsar return ConsumerBusy.
It could happen that pulsar didn't find the original consumer is dead
and recover takes some time.
fix #31631
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-01 15:23:19 +08:00
ba6d33cd57
fix: Support TLS for kafka connection ( #30468 )
...
#27977
Add extra configurations in milvus.yaml to pass certificates for kafka.
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-02-28 18:43:07 +08:00
e0f371c99b
fix: [skip e2e]use unique topic name for test pulsarCtl ( #30364 )
...
See also: #30363
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-01 11:05:04 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
f9508a4851
fix: Remove logging data when logging skip msg ( #29707 )
...
See also: #29696
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-15 11:40:51 +08:00
1f759837c4
fix: remove Unnecessary lock in config manager ( #29836 )
...
issue: #29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 13:48:49 +08:00
1c1f2a1371
enhance:change some logs ( #29579 )
...
related #29588
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
9e6da45497
fix: Use uber atomic instead sync/atomic which only supported after go v1.20 ( #29377 )
...
relate: https://github.com/milvus-io/milvus/issues/29376
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 16:06:49 +08:00
a0e1a1eb31
feat: support enable/disable mmap for index ( #29005 )
...
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
48f506b077
fix: Seperate msgstream initctx with lifecycle ctx ( #29308 )
...
The close of msgstream should be controlled by Close() not ctx from
outside when init clients
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-21 11:56:45 +08:00
46a937ac43
fix: msgstream memory leak caused by config event don't deregister ( #29266 )
...
relate: https://github.com/milvus-io/milvus/issues/28620
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 20:32:44 +08:00
de13865769
enhance: Add load/release partitions to replicate msg stream ( #28399 )
...
/kind improvement
issue: #25655
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-23 15:38:24 +08:00
3c3a57cb92
fix: Fix kafka config type error ( #28642 )
...
issue https://github.com/milvus-io/milvus/issues/28588
---------
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-23 14:36:22 +08:00
371875f650
Fix MQ Factory timeout unit test ( #28395 )
...
This PR change timeout case to a longer duration in case of slow runner
Also change timeout case behavior of using timeout deadline instead of waiting
See also #28394
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-13 21:44:18 +08:00
6749957e71
Refine RPC call in unwatch drop channel ( #27864 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 17:46:15 +08:00
086220dd15
Add the mock factory of msg stream ( #27834 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 20:14:08 +08:00
9b0ecbdca7
Support to replicate the mq message ( #27240 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
2b13078b14
Refine insert when send message ( #27491 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Cai Zhang <cai.zhang@zilliz.com>
2023-10-18 21:12:08 +08:00
a9982aa7c6
Remove topic empty restriction when preCreated topic ( #27351 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-10-10 18:53:32 +08:00
5d558623fe
Add revive sub-lints and fix existing problems ( #27495 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-07 20:53:38 +08:00
f962fc186f
Fix panic due to return nil pulsar client ( #27429 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 16:19:26 +08:00
c9653b1683
Add some log and improve TestSessionProcessActiveStandBy test case ( #27403 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-28 09:35:27 +08:00
7119cb29ca
Fix kafka consumer connection leak ( #27224 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 10:31:27 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
fe01d54eca
Set kafka read timeout to 10s and make it configurable ( #27238 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-20 22:01:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
791e6ef6c6
[Improvement] add pulsar metrics and fix timeout ( #26907 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-09-13 12:03:19 +08:00
45d9fb5929
Fix wrong error codes & names ( #26904 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 14:41:18 +08:00
1aa3a1e067
Refine `NewNmqClient` logic with nats.ConsumeDialer option ( #26987 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-11 20:17:17 +08:00
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek ( #26686 )
...
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
1a8cf5c415
Organize all mockery generation commands in Makefile ( #26826 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
d343888f3c
Fix pkg codecov & static-check script ( #26720 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 09:59:01 +08:00
ee7aef9272
Make pulsar request timeout configurable ( #26525 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-23 20:46:23 +08:00
85886f039e
[Fixup] msg context access data race when msgdispatcher ( #26463 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-21 19:16:22 +08:00
78bc688d16
Remove QueryMsgStream in MqFactory interface ( #26374 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-16 10:29:33 +08:00
4c4b2903f9
[Fixup] nats log configuration, open nats log at info level by deafult ( #26168 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-14 14:21:32 +08:00
a3c176045d
Fix kafka producer init many times ( #26314 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-13 16:59:30 +08:00
767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 ( #26179 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 10:05:14 +08:00
SimFG
congqixia
wei liu
jaime
Aldrin
Xiaofan
groot
XuanYang-cn
smellthemoon
aoiasd
yah01
Enwei Jiao
zhenshan.cao
yihao.dai
chyezh
yah01