milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated Monitoring and Alerting (markdown)

master
YiyunNi 2021-11-24 17:50:15 +08:00
parent 7f1a9831f5
commit b54c9cc923
1 changed files with 10 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
Monitoring-and-Alerting.md

@ -27,7 +27,7 @@ The ServiceMonitor Custom Resource Definition (CRD) enables you to declaratively
The following image illustrates Prometheus workflow.
![Prometheus_architecture](../../../../assets/prometheus_architecture.png)
![Prometheus_architecture](https://milvus.io/static/7cbbe2ab16b71ca69d405c889a90bdf6/f6a5a/prometheus_architecture.png)
## Prerequisites
@ -99,16 +99,16 @@ Download and import Milvus dashboard from the JSON file.
wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/monitor/grafana/milvus-dashboard.json
```
![Download_and_import](../../../../assets/import_dashboard.png)
![Download_and_import](https://milvus.io/static/ced62bc2eb07e9cbde6d1eb00c0fbd88/bbbf7/import_dashboard.png)
### 2. View metrics
Select the Milvus instance you want to monitor. Then you can see the Milvus components panel.
![Select_instance](../../../../assets/grafana_select.png)
![Select_instance](https://milvus.io/static/00a1238cd0f98b4337ea94777e8f4b46/bbbf7/grafana_select.png)
![Grafana_panel](../../../../assets/grafana_panel.png)
![Grafana_panel](https://milvus.io/static/bb8fa0d4fce3c044cb540a7bc7ff4f11/bbbf7/grafana_panel.png)
# Create an Alert for Milvus Services
@ -145,26 +145,26 @@ This tutorial assumes that you have Grafana installed and configured. If not, we
### 1. Add a new query
To add an alert for the memory usage of Milvus components, edit the Memory panel. Then, add a new query with the metric: `process_resident_memory_bytes{app_kubernetes_io_name="milvus", app_kubernetes_io_instance=~"my-release", namespace="default"}`
![Alert_metric](../../../../assets/alert_metric.png)
![Alert_metric](https://milvus.io/static/18103d1a30b5181fc6ee91800ea939f5/bbbf7/alert_metric.png)
### 2. Save the dashboard
Save the dashboard, and wait for a few minutes to see the alert.
![Alert_dashboard](../../../../assets/alert_dashboard.png)
![Alert_dashboard](https://milvus.io/static/fd5f1694aed3b5b5f29b639ad301a071/bbbf7/alert_dashboard.png)
Grafana alert query does not support template variables. Therefore, you should add a second query without any template variables in the labels. The second query is named as "A" by default. You can rename it by clicking on the dropdown.
![Alert_query](../../../../assets/alert_query.png)
![Alert_query](https://milvus.io/static/6ff9d7104bae4f442d79066d22dc7993/bbbf7/alert_query.png)
### 3. Add alert notifications
To receive alert notifications, add a "notification channel". Then, specify the channel in the field "Send to".
![Alert_notification](../../../../assets/alert_notification.png)
![Alert_notification](https://milvus.io/static/02677efc0008e03977429c0abd4838b0/1263b/alert_notification.png)
If the alert is successfully created and triggered, you will receive the notification as shown in the screenshot below.
![Notification_message](../../../../assets/notification_message.png)
![Notification_message](https://milvus.io/static/cab0cbc6f6979e44d8a2c77c66e9cd9a/bbbf7/notification_message.png)
To delete an alert, go to the "Alert" panel and click the delete button.
![Delete_alert](../../../../assets/delete_alert.png)
![Delete_alert](https://milvus.io/static/1a45b6cde7981603fc5b983d3d88c3ef/bbbf7/delete_alert.png)