milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated Monitoring and Alerting (markdown)

master
YiyunNi 2021-11-24 17:50:15 +08:00
parent 7f1a9831f5
commit b54c9cc923
1 changed files with 10 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
Monitoring-and-Alerting.md

@ -27,7 +27,7 @@ The ServiceMonitor Custom Resource Definition (CRD) enables you to declaratively
The following image illustrates Prometheus workflow. The following image illustrates Prometheus workflow.
![Prometheus_architecture](../../../../assets/prometheus_architecture.png) ![Prometheus_architecture](https://milvus.io/static/7cbbe2ab16b71ca69d405c889a90bdf6/f6a5a/prometheus_architecture.png)
## Prerequisites ## Prerequisites
@ -99,16 +99,16 @@ Download and import Milvus dashboard from the JSON file.
wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/monitor/grafana/milvus-dashboard.json wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/monitor/grafana/milvus-dashboard.json
``` ```
![Download_and_import](../../../../assets/import_dashboard.png) ![Download_and_import](https://milvus.io/static/ced62bc2eb07e9cbde6d1eb00c0fbd88/bbbf7/import_dashboard.png)
### 2. View metrics ### 2. View metrics
Select the Milvus instance you want to monitor. Then you can see the Milvus components panel. Select the Milvus instance you want to monitor. Then you can see the Milvus components panel.
![Select_instance](../../../../assets/grafana_select.png) ![Select_instance](https://milvus.io/static/00a1238cd0f98b4337ea94777e8f4b46/bbbf7/grafana_select.png)
![Grafana_panel](../../../../assets/grafana_panel.png) ![Grafana_panel](https://milvus.io/static/bb8fa0d4fce3c044cb540a7bc7ff4f11/bbbf7/grafana_panel.png)
# Create an Alert for Milvus Services # Create an Alert for Milvus Services
@ -145,26 +145,26 @@ This tutorial assumes that you have Grafana installed and configured. If not, we
### 1. Add a new query ### 1. Add a new query
To add an alert for the memory usage of Milvus components, edit the Memory panel. Then, add a new query with the metric: `process_resident_memory_bytes{app_kubernetes_io_name="milvus", app_kubernetes_io_instance=~"my-release", namespace="default"}` To add an alert for the memory usage of Milvus components, edit the Memory panel. Then, add a new query with the metric: `process_resident_memory_bytes{app_kubernetes_io_name="milvus", app_kubernetes_io_instance=~"my-release", namespace="default"}`
![Alert_metric](../../../../assets/alert_metric.png) ![Alert_metric](https://milvus.io/static/18103d1a30b5181fc6ee91800ea939f5/bbbf7/alert_metric.png)
### 2. Save the dashboard ### 2. Save the dashboard
Save the dashboard, and wait for a few minutes to see the alert. Save the dashboard, and wait for a few minutes to see the alert.
![Alert_dashboard](../../../../assets/alert_dashboard.png) ![Alert_dashboard](https://milvus.io/static/fd5f1694aed3b5b5f29b639ad301a071/bbbf7/alert_dashboard.png)
Grafana alert query does not support template variables. Therefore, you should add a second query without any template variables in the labels. The second query is named as "A" by default. You can rename it by clicking on the dropdown. Grafana alert query does not support template variables. Therefore, you should add a second query without any template variables in the labels. The second query is named as "A" by default. You can rename it by clicking on the dropdown.
![Alert_query](../../../../assets/alert_query.png) ![Alert_query](https://milvus.io/static/6ff9d7104bae4f442d79066d22dc7993/bbbf7/alert_query.png)
### 3. Add alert notifications ### 3. Add alert notifications
To receive alert notifications, add a "notification channel". Then, specify the channel in the field "Send to". To receive alert notifications, add a "notification channel". Then, specify the channel in the field "Send to".
![Alert_notification](../../../../assets/alert_notification.png) ![Alert_notification](https://milvus.io/static/02677efc0008e03977429c0abd4838b0/1263b/alert_notification.png)
If the alert is successfully created and triggered, you will receive the notification as shown in the screenshot below. If the alert is successfully created and triggered, you will receive the notification as shown in the screenshot below.
![Notification_message](../../../../assets/notification_message.png) ![Notification_message](https://milvus.io/static/cab0cbc6f6979e44d8a2c77c66e9cd9a/bbbf7/notification_message.png)
To delete an alert, go to the "Alert" panel and click the delete button. To delete an alert, go to the "Alert" panel and click the delete button.
![Delete_alert](../../../../assets/delete_alert.png) ![Delete_alert](https://milvus.io/static/1a45b6cde7981603fc5b983d3d88c3ef/bbbf7/delete_alert.png)