Monitor your Dedicated cluster (#5409)
* scaffolding for clustered grafana dashboard * add descriptions of dedicated monitoring dashboard cells * Apply suggestions from code review Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com> * Update content/influxdb/cloud-dedicated/admin/monitor-your-cluster.md --------- Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com>pull/5402/head^2
Scott Anderson
GitHub
parent
2aac3cc2e8
commit
aa1732d0d2
|
|
@ -0,0 +1,390 @@
|
||||||
|
---
|
||||||
|
title: Monitor your cluster
|
||||||
|
seotitle: Monitor your InfluxDB Cloud Dedicated cluster
|
||||||
|
description: >
|
||||||
|
Use the Grafana dashboard provided by InfluxData to monitor your
|
||||||
|
InfluxDB Cloud Dedicated cluster.
|
||||||
|
menu:
|
||||||
|
influxdb_cloud_dedicated:
|
||||||
|
parent: Administer InfluxDB Cloud
|
||||||
|
weight: 104
|
||||||
|
---
|
||||||
|
|
||||||
|
Use the Grafana dashboard provided by InfluxData to monitor your
|
||||||
|
{{< product-name >}} cluster.
|
||||||
|
|
||||||
|
{{% note %}}
|
||||||
|
#### Not available for all clusters
|
||||||
|
|
||||||
|
{{< product-name >}} monitoring dashboards are not available for all clusters.
|
||||||
|
For questions about availability, [contact InfluxData support](https://support.influxdata.com).
|
||||||
|
{{% /note %}}
|
||||||
|
|
||||||
|
- [Access your monitoring dashboard](#access-your-monitoring-dashboard)
|
||||||
|
- [Dashboard sections and cells](#dashboard-sections-and-cells)
|
||||||
|
|
||||||
|
{{< img-hd src="/img/influxdb/clustered-admin-monitoring-dashboard.png" alt="InfluxDB Cloud Dedicated monitoring dashboard" />}}
|
||||||
|
|
||||||
|
## Access your monitoring dashboard
|
||||||
|
|
||||||
|
To access your {{< product-name >}} monitoring dashboard, visit the
|
||||||
|
`/observability` endpoint of your {{< product-name >}} cluster in your browser:
|
||||||
|
|
||||||
|
<pre>
|
||||||
|
<a href="https://{{< influxdb/host >}}/observability">https://{{< influxdb/host >}}/observability</a>
|
||||||
|
</pre>
|
||||||
|
|
||||||
|
Use the credentials provided by InfluxData to log into your cluster monitoring dashboard.
|
||||||
|
If you do not have login credentials, [contact InfluxData support](https://support.influxdata.com).
|
||||||
|
|
||||||
|
## Dashboard sections and cells
|
||||||
|
|
||||||
|
The dashboard is divided into the following sections that visualize metrics
|
||||||
|
related to the health of components in your {{< product-name >}} cluster:
|
||||||
|
|
||||||
|
- [Query Tier Cpu/Mem](#query-tier-cpumem)
|
||||||
|
- [Query Tier](#query-tier)
|
||||||
|
- [Ingest Tier Cpu/Mem](#ingest-tier-cpumem)
|
||||||
|
- [Ingest Tier](#ingest-tier)
|
||||||
|
- [Compaction Tier Cpu/Mem](#compaction-tier-cpumem)
|
||||||
|
- [Compactor](#compactor)
|
||||||
|
- [Ingestor Catalog Operations](#ingestor-catalog-operations)
|
||||||
|
- [Catalog Operations Overview](#catalog-operations-overview)
|
||||||
|
|
||||||
|
### Query Tier Cpu/Mem
|
||||||
|
|
||||||
|
The **Query Tier Cpu/Mem** section displays the CPU and memory usage of query
|
||||||
|
pods as reported by Kubernetes.
|
||||||
|
[Queriers](/influxdb/cloud-dedicated/reference/internals/storage-engine/#querier)
|
||||||
|
handle query requests and returns query results for requests.
|
||||||
|
|
||||||
|
- [CPU Utilization (k8s)](#cpu-utilization-k8s)
|
||||||
|
- [Memory Usage (k8s)](#memory-usage-k8s)
|
||||||
|
|
||||||
|
#### CPU Utilization (k8s)
|
||||||
|
|
||||||
|
The CPU utilization of query pods as reported by the Kubernetes container usage.
|
||||||
|
Usage is reported by the number of CPU cores used by pods, including
|
||||||
|
fractional cores.
|
||||||
|
The CPU limit is represented by the top line in the visualization.
|
||||||
|
|
||||||
|
#### Memory Usage (k8s)
|
||||||
|
|
||||||
|
The memory usage of the query pod containers per cgroup as reported by Kubernetes.
|
||||||
|
Usage is reported in a magnitude of bytes.
|
||||||
|
The memory limit is represented by the top line in the visualization.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Query Tier
|
||||||
|
|
||||||
|
The **Query Tier** section displays metrics reported from the InfluxDB gRPC
|
||||||
|
query API.
|
||||||
|
[Queriers](/influxdb/cloud-dedicated/reference/internals/storage-engine/#querier)
|
||||||
|
handle query requests and returns query results for requests.
|
||||||
|
|
||||||
|
- [gRPC Requests (ok)](#grpc-requests-ok)
|
||||||
|
- [gRPC Requests (not ok)](#grpc-requests-not-ok)
|
||||||
|
- [Request Duration (flight DoGet) (ok + !ok)](#request-duration-flight-doget-ok--ok)
|
||||||
|
- [Successful Request Duration (flight DoGet)](#successful-request-duration-flight-doget)
|
||||||
|
- [Acquire Duration](#acquire-duration)
|
||||||
|
|
||||||
|
#### gRPC Requests (ok)
|
||||||
|
|
||||||
|
The rate of gRPC requests for different endpoints that returned the `OK` status code,
|
||||||
|
summed across all queriers.
|
||||||
|
Request rate is reported in requests per second.
|
||||||
|
|
||||||
|
#### gRPC Requests (not ok)
|
||||||
|
|
||||||
|
The rate of gRPC requests for all endpoints that returned a status code other
|
||||||
|
than `OK`, summed across all queriers.
|
||||||
|
Request rate is reported in requests per second.
|
||||||
|
|
||||||
|
#### Request Duration (flight DoGet) (ok + !ok)
|
||||||
|
|
||||||
|
A gRPC request duration heatmap for all requests to the `DoGet` endpoint
|
||||||
|
regardless of request status.
|
||||||
|
|
||||||
|
The heatmap shows how many requests occurred in each duration "bucket" per time
|
||||||
|
interval and provides insight into how long a typical query request takes.
|
||||||
|
It also shows, at a glance, the predominate latency range as well as the
|
||||||
|
minimum and maximum durations of all query requests.
|
||||||
|
|
||||||
|
The color scheme is a indicator of the value of each cell relative to the
|
||||||
|
_currently displayed data_.
|
||||||
|
|
||||||
|
#### Successful Request Duration (flight DoGet)
|
||||||
|
|
||||||
|
A gRPC request duration heatmap for successful requests to the `DoGet` endpoint.
|
||||||
|
|
||||||
|
The heatmap shows how many requests occurred in each duration "bucket" per time
|
||||||
|
interval and provides insight into how long a typical successful query request takes.
|
||||||
|
It also shows, at a glance, the predominate latency range as well as the
|
||||||
|
minimum and maximum durations of successful query requests.
|
||||||
|
|
||||||
|
The color scheme is a indicator of the value of each cell relative to the
|
||||||
|
_currently displayed data_.
|
||||||
|
|
||||||
|
#### Acquire Duration
|
||||||
|
|
||||||
|
A heatmap of how long a query waits to pass the query _semaphore_--a mechanism
|
||||||
|
that limits the number of concurrent query requests that can be processed and
|
||||||
|
protects against Out of Memory (OOM) errors that can be caused by unaccounted-for
|
||||||
|
data structures that may occur during query planning and execution.
|
||||||
|
This cell only provides information about the queries waiting for the semaphore,
|
||||||
|
not the time holding it.
|
||||||
|
|
||||||
|
This cell can be used to gauge how much query latency is added due to a high
|
||||||
|
cluster load.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Ingest Tier Cpu/Mem
|
||||||
|
|
||||||
|
The **Query Tier Cpu/Mem** section displays the CPU and memory usage of Ingester
|
||||||
|
pods as reported by Kubernetes.
|
||||||
|
[Ingesters](/influxdb/cloud-dedicated/reference/internals/storage-engine/#ingester)
|
||||||
|
process line protocol submitted in write requests and persist time series data
|
||||||
|
to the [Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store).
|
||||||
|
|
||||||
|
- [CPU Utilization Ingesters (k8s)](#cpu-utilization-ingesters-k8s)
|
||||||
|
- [Memory Usage Ingesters (k8s)](#memory-usage-ingesters-k8s)
|
||||||
|
- [CPU Utilization Routers (k8s)](#cpu-utilization-routers-k8s)
|
||||||
|
- [Memory Usage Routers (k8s)](#memory-usage-routers-k8s)
|
||||||
|
|
||||||
|
#### CPU Utilization Ingesters (k8s)
|
||||||
|
|
||||||
|
CPU Utilization of Ingester pods as reported by the Kubernetes container usage.
|
||||||
|
Usage is reported by the number of CPU cores used by pods, including
|
||||||
|
fractional cores.
|
||||||
|
The CPU limit is represented by the top line in the visualization.
|
||||||
|
|
||||||
|
#### Memory Usage Ingesters (k8s)
|
||||||
|
|
||||||
|
Memory usage of the Ingester pod containers per cgroup as reported by Kubernetes.
|
||||||
|
Usage is reported in a magnitude of bytes.
|
||||||
|
The memory limit is represented by the top line in the visualization.
|
||||||
|
|
||||||
|
#### CPU Utilization Routers (k8s)
|
||||||
|
|
||||||
|
CPU utilization of Ingester router pods as reported by the Kubernetes container usage.
|
||||||
|
Usage is reported by the number of CPU cores used by pods, including
|
||||||
|
fractional cores.
|
||||||
|
|
||||||
|
#### Memory Usage Routers (k8s)
|
||||||
|
|
||||||
|
Memory usage of the Ingester router pod containers per cgroup as reported by Kubernetes.
|
||||||
|
Usage is reported in a magnitude of bytes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Ingest Tier
|
||||||
|
|
||||||
|
The **Ingest Tier** section displays metrics reported from the InfluxDB gRPC
|
||||||
|
and HTTP write APIs.
|
||||||
|
[Ingesters](/influxdb/cloud-dedicated/reference/internals/storage-engine/#ingester)
|
||||||
|
process line protocol submitted in write requests and persist time series data
|
||||||
|
to the [Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store).
|
||||||
|
|
||||||
|
- [Write Requests (at router)](#write-requests-at-router)
|
||||||
|
- [LP Ingest (at router)](#lp-ingest-at-router-lines)
|
||||||
|
<em class="op50">(lines)</em>
|
||||||
|
- [LP Ingest (at router)](#lp-ingest-at-router-bytes)
|
||||||
|
<em class="op50">(bytes)</em>
|
||||||
|
- [HTTP request error rate (server's POV at Router)](#http-request-error-rate-server's-pov-at-router)
|
||||||
|
- [Healthy Upstream Ingesters per Router](#healthy-upstream-ingesters-per-router)
|
||||||
|
- [Persist Queue Depth](#persist-queue-depth)
|
||||||
|
- [Persist Task Queue Duration](#persist-task-queue-duration)
|
||||||
|
- [Ingester Disk Data Directory Usage](#ingester-disk-data-directory-usage)
|
||||||
|
- [Ingest Blocked Time (24h)](#ingest-blocked-time-24h)
|
||||||
|
- [Max Persist Queue Depth](#max-persist-queue-depth)
|
||||||
|
- [Write Logs (10 examples)](#write-logs-10-examples)
|
||||||
|
|
||||||
|
#### Write Requests (at router)
|
||||||
|
|
||||||
|
Number of write operations completed across all Ingester routers.
|
||||||
|
Requests are grouped by state (success or error).
|
||||||
|
Request rate is reported in requests per second.
|
||||||
|
|
||||||
|
#### LP Ingest (at router) {#lp-ingest-at-router-lines metadata="lines"}
|
||||||
|
|
||||||
|
Rate of lines of line protocol being received by each router and across all
|
||||||
|
Ingester routers.
|
||||||
|
Request rate is reported in lines per second.
|
||||||
|
|
||||||
|
#### LP Ingest (at router) {#lp-ingest-at-router-bytes metadata="bytes"}
|
||||||
|
|
||||||
|
Rate of bytes of line protocol being received by each router and across all
|
||||||
|
Ingester routers.
|
||||||
|
Request rate is reported in bytes per second.
|
||||||
|
|
||||||
|
#### HTTP request error rate (server's POV at Router)
|
||||||
|
|
||||||
|
HTTP request error rate reported by the InfluxDB v3 HTTP request handler.
|
||||||
|
Error rate is represented the percentage in total requests that return a non-2xx
|
||||||
|
response code.
|
||||||
|
|
||||||
|
#### Healthy Upstream Ingesters per Router
|
||||||
|
|
||||||
|
The number of healthy upstream Ingesters each router detects.
|
||||||
|
This reflects the router's RPC request balancer or circuit breaker state.
|
||||||
|
|
||||||
|
This can indicate when routers can't connect to Ingesters becaåuse of because of
|
||||||
|
issues in the ingest pipeline such as network issues or Ingester availability.
|
||||||
|
|
||||||
|
The Persist Queue is the queue for persisting, or saving to s3, new parquey files.
|
||||||
|
|
||||||
|
#### Persist Queue Depth
|
||||||
|
|
||||||
|
The number of queued persist jobs that have not started.
|
||||||
|
Each persist jobs consists of taking data from the Write Ahead Log (WAL),
|
||||||
|
storing it in a Parquet file, and saving the Parquet file to the
|
||||||
|
[Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store).
|
||||||
|
|
||||||
|
If the persist queue is growing it means Ingesters are not keeping up with the
|
||||||
|
incoming write load and may result in Ingester failure.
|
||||||
|
|
||||||
|
#### Persist Task Queue Duration
|
||||||
|
|
||||||
|
A heatmap that shows the time persist jobs spend in the queue before being executed.
|
||||||
|
|
||||||
|
Longer queue times indicate slower persist job execution times which may be due
|
||||||
|
to network or internal resource constraints, or an increasing
|
||||||
|
[queue depth](#persist-queue-depth).
|
||||||
|
|
||||||
|
#### Ingester Disk Data Directory Usage
|
||||||
|
|
||||||
|
The per-pod disk usage as a percentage of the Ingesters' data directory.
|
||||||
|
The WAL is stored on a disk attached to the Ingesters.
|
||||||
|
As the WAL grows, more disk space is used.
|
||||||
|
If Ingesters run out of disk, the WAL stops functioning.
|
||||||
|
|
||||||
|
#### Ingest Blocked Time (24h)
|
||||||
|
|
||||||
|
The amount of time the ingest pipeline has been marked as saturated and
|
||||||
|
rejected write requests.
|
||||||
|
|
||||||
|
#### Max Persist Queue Depth
|
||||||
|
|
||||||
|
The queue depth as a percentage of the configured maximum queue depth.
|
||||||
|
This shows the saturation level of the most saturated Ingester.
|
||||||
|
Once the maximum queue depth is reached, writes are rejected.
|
||||||
|
|
||||||
|
#### Write Logs (10 examples)
|
||||||
|
|
||||||
|
A sample of 10 write logs from the displayed time period.
|
||||||
|
_These do not represent the most recent logs._
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Compaction Tier Cpu/Mem
|
||||||
|
|
||||||
|
The **Compaction Tier Cpu/Mem** section displays the CPU and memory usage of
|
||||||
|
Compactor pods as reported by Kubernetes.
|
||||||
|
[Compactors](/influxdb/cloud-dedicated/reference/internals/storage-engine/#compactor)
|
||||||
|
process and compress parquet files in the
|
||||||
|
[Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store)
|
||||||
|
to continually optimize storage.
|
||||||
|
|
||||||
|
- [CPU Utilization (k8s)](#compaction-cpu-utilization)
|
||||||
|
- [Memory Usage (k8s)](#compaction-memory-usage)
|
||||||
|
|
||||||
|
#### CPU Utilization (k8s) {#compaction-cpu-utilization}
|
||||||
|
|
||||||
|
The CPU utilization of compactor pods as reported by the Kubernetes container usage.
|
||||||
|
Usage is reported by the number of CPU cores used by pods, including
|
||||||
|
fractional cores.
|
||||||
|
The CPU limit is represented by the top line in the visualization.
|
||||||
|
|
||||||
|
#### Memory Usage (k8s) {#compaction-memory-usage}
|
||||||
|
|
||||||
|
The memory usage of compactor pod containers per cgroup as reported by Kubernetes.
|
||||||
|
Usage is reported in a magnitude of bytes.
|
||||||
|
The memory limit is represented by the top line in the visualization.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Compactor
|
||||||
|
|
||||||
|
The **Compactor** section displays metrics related to the compaction of Parquet
|
||||||
|
files in the [Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store).
|
||||||
|
[Compactors](/influxdb/cloud-dedicated/reference/internals/storage-engine/#compactor)
|
||||||
|
process and compress Parquet files to continually optimize storage.
|
||||||
|
|
||||||
|
- [Compactor: L0 File Counts (5m bucket width)](#compactor-l0-file-counts-5m-bucket-width)
|
||||||
|
|
||||||
|
#### Compactor: L0 File Counts (5m bucket width)
|
||||||
|
|
||||||
|
A histogram of the quantity of L0-compacted files at time of compaction.
|
||||||
|
|
||||||
|
Ingesters create Parquet files using L0 (level zero) compaction.
|
||||||
|
As Compactors process and compact Parquet files over time, they do so in the
|
||||||
|
following levels:
|
||||||
|
|
||||||
|
- **L0**: Uncompacted
|
||||||
|
- **L1**: 4 L0 files compacted together
|
||||||
|
- **L2**: 4 L1 files compacted together
|
||||||
|
- **L3**: 4 L2 files compacted together
|
||||||
|
|
||||||
|
Parquet files store data partitioned by time and optionally tags
|
||||||
|
_(see [Manage data partition](https://docs.influxdata.com/influxdb/cloud-dedicated/admin/custom-partitions/))_.
|
||||||
|
After four L0 files accumulate for a partition, they are are eligible for compaction.
|
||||||
|
If the compactor is keeping up with the incoming write load, all compaction
|
||||||
|
events will have exactly four files. If the number of L0 files compacted begins to
|
||||||
|
to increase, it indicates the compactor is not keeping up.
|
||||||
|
|
||||||
|
This histogram helps to determine if the Compactor is starting compactions as
|
||||||
|
soon as it can.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Ingestor Catalog Operations
|
||||||
|
|
||||||
|
The **Ingestor Catalog Operations** section displays metrics related to
|
||||||
|
Catalog operations requested by Ingesters.
|
||||||
|
The [Catalog](/influxdb/cloud-dedicated/reference/internals/storage-engine/#catalog)
|
||||||
|
is a relational database that stores metadata related to your time series data
|
||||||
|
including schema information and physical locations of partitions in the
|
||||||
|
[Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store).
|
||||||
|
|
||||||
|
- [Catalog Ops - success](#catalog-ops---success)
|
||||||
|
- [Catalog Ops - error](#catalog-ops---error)
|
||||||
|
- [Catalog Op Latency (P90)](#catalog-op-latency-p90)
|
||||||
|
|
||||||
|
#### Catalog Ops - success
|
||||||
|
|
||||||
|
The rate of successful Catalog operations per second requested by Ingesters.
|
||||||
|
Higher rates of successful Catalog operations requested by Ingesters indicate
|
||||||
|
a high write load.
|
||||||
|
|
||||||
|
#### Catalog Ops - error
|
||||||
|
|
||||||
|
The rate of erred catalog operations per second requested by Ingesters.
|
||||||
|
Higher rates of erred Catalog operations requested by Ingesters indicate
|
||||||
|
that the Catalog may be overloaded or unresponsive.
|
||||||
|
|
||||||
|
#### Catalog Op Latency (P90)
|
||||||
|
|
||||||
|
The 90th percentile (P90) of query latency against the catalog service per operation.
|
||||||
|
A high P90 value indicates that the Catalog may be overloaded.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Catalog Operations Overview
|
||||||
|
|
||||||
|
The **Catalog Operations Overview** section displays metrics related to
|
||||||
|
Catalog operations requested by all components of your {{< product-name >}} cluster.
|
||||||
|
|
||||||
|
- [Requests per Operation - success](#requests-per-operation---success)
|
||||||
|
- [Requests per Operation - error](#requests-per-operation---error)
|
||||||
|
|
||||||
|
#### Requests per Operation - success
|
||||||
|
|
||||||
|
The rate of successful Catalog requests per second by operation.
|
||||||
|
|
||||||
|
#### Requests per Operation - error
|
||||||
|
|
||||||
|
The rate of erred Catalog requests per second by operation.
|
||||||
|
Higher rates of erred Catalog operations indicate that the Catalog may be
|
||||||
|
overloaded or unresponsive.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 139 KiB |
Loading…
Reference in New Issue