Commit Graph

396 Commits (10kcp)

Author SHA1 Message Date
yihao.dai df4d5e1096
enhance: [10kcp] Read metadata concurrently to accelerate recovery (#38404)
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38403

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-12 16:39:06 +08:00
Zhen Ye 99279e0bef
enhance: remove the rpc layer of coordinator when enabling standalone or mixcoord (#38246)
issue: #33285
pr: #37815

- remove the rpc layer of coordinator when enabling standalone or
mixcoord
- move health check into init

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-05 17:27:53 +08:00
yihao.dai 312475d1f1
enhance: [10kcp] remove the rpc level of coordinator (#37984)
issue: https://github.com/milvus-io/milvus/issues/37764

- add a local client to call local server directly for
querycoord/rootcoord/datacoord.
- enable local client if milvus is running mixcoord or standalone mode.

Signed-off-by: chyezh <chyezh@outlook.com>

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Zhen Ye <chyezh@outlook.com>
2024-11-25 14:50:42 +08:00
yihao.dai 4845e4d679
enhance: [10kcp] Revert "enhance: remove the rpc level of coordinator (#37914)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-21 21:35:29 +08:00
Zhen Ye 1a6b98be77
enhance: remove the rpc level of coordinator (#37876)
issue: #33285
pr: #37722

- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-21 15:21:11 +08:00
yihao.dai 9cb5396cf6
enhance: Use common gc config (#36668) (#36670)
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

pr: https://github.com/milvus-io/milvus/pull/36668

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 19:49:20 +08:00
Zhen Ye e34fa0461b
fix: port listen racing in mix or standalone mode (#36459)
issue: #36441
pr: #36442

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:01:15 +08:00
wei liu 14fe3213b1
fix: Revert "enhance: avoid the coexistence of the old coordinator and the new node/proxy (#35760)" (#36114)
issue: #36100
pr: #36113
This reverts commit 50ec3dd4db.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-09 17:57:06 +08:00
SimFG d3bf7a2d27
fix: [2.4] delay to start the metric server port (#36085)
- issue: #36083
- pr: #36080
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-09 10:11:06 +08:00
congqixia 55b33cd3cf
fix: [2.4] Fix tracing config update logic (#35928) (#35998)
Cherry-pick from master
pr: #35928 
Related to #35927

There are serveral issue this PR addresses:
- Use `ResetTraceConfig` method instead init one in update event handler
- Implement dynamic stats.Handler to receive tracing config update event
- Update `enable_trace` flag when `ResetTraceConfig` is invoked
- Change `enable_trace` to `std::atomic<bool>` in case of data race

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-06 11:19:05 +08:00
wei liu 50ec3dd4db
enhance: avoid the coexistence of the old coordinator and the new node/proxy (#35760)
issue: #35719
pr: #35720 
In standalone mode, block the start process until the new coordinator is
active to avoid the coexistence of the old coordinator and the new
node/proxy
1. In the start/restart process, the new coordinator will become active
immediately and will not be blocked
2. In the rolling upgrade process, the new coordinator will not be
active until the old coordinator is down, and it will be blocked

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-29 12:33:01 +08:00
Zhen Ye 7062e6fcbf
enhance: enable asan for milvus (#35687)
issue: #35626
pr: #35627

Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-26 09:54:58 +08:00
SimFG 5b5119a51f
feat: [2.4] provide more general configuration to control mmap behavior (#35609)
- issue: #35273
- pr: #35359

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-23 12:35:02 +08:00
wei liu e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 (#34394) (#35555)
issue: #34252
pr: #34394 #35072 #35084

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
Ted Xu 57d4bcbf15
enhance: adding the msgchannel section in generated yaml (#35466)
See #32168

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-14 19:03:11 +08:00
Ted Xu ce53e79f12
fix: enable milvus.yaml check (#34567) (#35446)
See #32168

pr: #34567 #35152

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-13 19:00:23 +08:00
Jiquan Long e598233b6a
feat: put inverted index into ram (#35222) (#35223)
fix: https://github.com/milvus-io/milvus/issues/35224
pr: https://github.com/milvus-io/milvus/pull/35222

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-06 10:08:16 +08:00
wei liu f8f2736cda
enhance: Skip manual stopped component during health check (#34953) (#35123)
pr: #34953
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.

to solve this, we make `healthz` API skip the manual stopped component.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-31 18:01:51 +08:00
cai.zhang 74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time (#34950)
issue: #34703 

master pr: #34949

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
SimFG 0e226502e4
enhance: [2.4] pick default root password and log level pr (#34777)
default root password
- issue: #33058
- pr: #34752

set log level
- issue: #34756
- pr: #34757

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-18 13:45:43 +08:00
jaime 6423b6c718
enhance: move rocksmq from internal to pkg (#34165)
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 13:36:05 +08:00
wei liu 4513569207
enhance: add restful api to trigger component stop (#32076) (#33799)
issue: #32698
pr: #32076
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 21:53:59 +08:00
aoiasd aaf6c85095
fix: paramtable cache cause dynamic config non-dynamic (#33473) (#33590)
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/33473

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 10:15:51 +08:00
Bingyi Sun 4724779b3b
enhance: remove fallback keys for config generator (#32946)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-13 13:33:31 +08:00
aoiasd 54a51b1236
enhance: Support dynamic config for opentelemetry trace (#32169)
relate: https://github.com/milvus-io/milvus/issues/31940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-09 17:43:30 +08:00
Ted Xu 78d32bd8b2
enhance: update milvus.yaml (#31832)
See #32168

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-04-16 16:17:19 +08:00
shaoyue be53e028a8
doc: [skip e2e] add config-docs-generator (#22134)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Part of issue: #21856

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2024-04-03 14:13:13 +08:00
Xiaofan b6fefee0cf
fix: etcd not connectable when auth enabled (#31633)
Fix etcd config source didn't respect auth enabled
Also removed pulsar recoverable error when pulsar return ConsumerBusy.
It could happen that pulsar didn't find the original consumer is dead
and recover takes some time.
fix #31631

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-01 15:23:19 +08:00
Bingyi Sun 0ac9bb4a9c
enhance: add mmap migration tool (#30909)
issue: #30908

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:51:09 +08:00
chyezh 0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317)
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
smellthemoon 9815cf50c9
fix: close rocksmq and natsmq when shutdown (#30706)
related: #30703 #30355

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-26 11:38:55 +08:00
chyezh 211143c5e6
enhance: add basic information of milvus into metrics (#29665)
add basic build information and runtime component dependency into
metrics.

issue: #29664

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:47:02 +08:00
congqixia 20a3569c14
enhance: Set correct role for non-standalone deployment (#30303)
See also #30211

After fix initialization problem, distributed components do no have
their role set. This will cause logger & tracing miss component service
info when recording information.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-26 14:05:01 +08:00
PowderLi 08ca0a2ca5
feat: support etcd authentication (#30226)
issue: #28895
add 3 configuration for ETCD config

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-24 11:35:00 +08:00
congqixia c9860e3d2a
fix: Init paramtable after milvus role setup (#30211)
See also #30176

Move paramtable.Init after env setup in roles.Run. Also introduced a
flag for mixture run to set role correctly for mixture mode.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-23 19:10:56 +08:00
Bingyi Sun dad27cbb40
fix: fix wrong log file name when running as mixture type (#29979)
See also: #25323, #29969
many users reported log file name is incorrect when starting in mixture
type.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-19 11:48:54 +08:00
Jiquan Long a3bae80b59
enhance: print total memory when milvus starts (#29351)
fix: #29349

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-07 20:09:49 +08:00
SimFG dd9c61831d
enhance: Support to get the param value in the runtime (#29297)
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 18:36:44 +08:00
Bingyi Sun e8c5e75786
enhance: ignore proto generated files (#28537)
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-22 10:20:22 +08:00
jaime ec0d9d8fd4
Fix delete session key with prefix (#28260)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-08 15:22:21 +08:00
wei liu 0c69f48ba4
refine stop order (#28016)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-01 11:40:15 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 12:18:15 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069)
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-10-27 07:36:12 +08:00
wayblink aa1fba79b1
Register metrics before component start (#27744)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-10-19 17:14:10 +08:00
jaime ac2d1bb5c2
Support receive signals from parent process (#27756)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-18 20:20:11 +08:00
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-16 10:24:10 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
congqixia f7b2ad6650
Fix component pointer to `component` interface never be nil caused panic (#27072)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-14 13:37:18 +08:00