Add the blog post for the beta graduation of HPA ContainerResource type metric (#39822)
* Add the blog post for the beta graduation of HPA ContainerResource type metric * fix based on reviews * fix based on the suggestion * fix based on review * update date * Apply changes from code review Co-authored-by: Nate W. <natew@cncf.io> * Push back publication date and timezone --------- Co-authored-by: Tim Bannister <tim@scalefactory.com> Co-authored-by: Nate W. <natew@cncf.io>pull/38834/head
Kensei Nakada
GitHub
parent
f787489ec5
commit
73437c2d98
|
|
@ -0,0 +1,108 @@
|
|||
---
|
||||
layout: blog
|
||||
title: "Kubernetes 1.27: HorizontalPodAutoscaler ContainerResource type metric moves to beta"
|
||||
date: 2023-05-02T12:00:00+0800
|
||||
slug: hpa-container-resource-metric
|
||||
---
|
||||
|
||||
**Author:** [Kensei Nakada](https://github.com/sanposhiho) (Mercari)
|
||||
|
||||
Kubernetes 1.20 introduced the [`ContainerResource` type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics)
|
||||
in HorizontalPodAutoscaler (HPA).
|
||||
|
||||
In Kubernetes 1.27, this feature moves to beta and the corresponding feature gate (`HPAContainerMetrics`) gets enabled by default.
|
||||
|
||||
## What is the ContainerResource type metric
|
||||
|
||||
The ContainerResource type metric allows us to configure the autoscaling based on resource usage of individual containers.
|
||||
|
||||
In the following example, the HPA controller scales the target
|
||||
so that the average utilization of the cpu in the application container of all the pods is around 60%.
|
||||
(See [the algorithm details](/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details)
|
||||
to know how the desired replica number is calculated exactly)
|
||||
|
||||
```yaml
|
||||
type: ContainerResource
|
||||
containerResource:
|
||||
name: cpu
|
||||
container: application
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 60
|
||||
```
|
||||
|
||||
## The difference from the Resource type metric
|
||||
|
||||
HPA already had a [Resource type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-resource-metrics).
|
||||
|
||||
You can define the target resource utilization like the following,
|
||||
and then HPA will scale up/down the replicas based on the current utilization.
|
||||
|
||||
```yaml
|
||||
type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 60
|
||||
```
|
||||
|
||||
But, this Resource type metric refers to the average utilization of the **Pods**.
|
||||
|
||||
In case a Pod has multiple containers, the utilization calculation would be:
|
||||
|
||||
```
|
||||
sum{the resource usage of each container} / sum{the resource request of each container}
|
||||
```
|
||||
|
||||
The resource utilization of each container may not have a direct correlation or may grow at different rates as the load changes.
|
||||
|
||||
For example:
|
||||
- A sidecar container is only providing an auxiliary service such as log shipping.
|
||||
If the application does not log very frequently or does not produce logs in its hotpath
|
||||
then the usage of the log shipper will not grow.
|
||||
- A sidecar container which provides authentication. Due to heavy caching
|
||||
the usage will only increase slightly when the load on the main container increases.
|
||||
In the current blended usage calculation approach this usually results in
|
||||
the HPA not scaling up the deployment because the blended usage is still low.
|
||||
- A sidecar may be injected without resources set which prevents scaling
|
||||
based on utilization. In the current logic the HPA controller can only scale
|
||||
on absolute resource usage of the pod when the resource requests are not set.
|
||||
|
||||
And, in such case, if only one container's resource utilization goes high,
|
||||
the Resource type metric may not suggest scaling up.
|
||||
|
||||
So, for the accurate autoscaling, you may want to use the ContainerResource type metric for such Pods instead.
|
||||
|
||||
## What's new for the beta?
|
||||
|
||||
For Kubernetes v1.27, the ContainerResource type metric is available by default as described at the beginning
|
||||
of this article.
|
||||
(You can still disable it by the `HPAContainerMetrics` feature gate.)
|
||||
|
||||
Also, we've improved the observability of HPA controller by exposing some metrics from the kube-controller-manager:
|
||||
- `metric_computation_total`: Number of metric computations.
|
||||
- `metric_computation_duration_seconds`: The time that the HPA controller takes to calculate one metric.
|
||||
- `reconciliations_total`: Number of reconciliation of HPA controller.
|
||||
- `reconciliation_duration_seconds`: The time that the HPA controller takes to reconcile a HPA object once.
|
||||
|
||||
These metrics have labels `action` (`scale_up`, `scale_down`, `none`) and `error` (`spec`, `internal`, `none`).
|
||||
And, in addition to them, the first two metrics have the `metric_type` label
|
||||
which corresponds to `.spec.metrics[*].type` for a HorizontalPodAutoscaler.
|
||||
|
||||
All metrics are useful for general monitoring of HPA controller,
|
||||
you can get deeper insight into which part has a problem, where it takes time, how much scaling tends to happen at which time on your cluster etc.
|
||||
|
||||
Another minor stuff, we've changed the `SuccessfulRescale` event's messages
|
||||
so that everyone can check whether the events came from the resource metric or
|
||||
the container resource metric (See [the related PR](https://github.com/kubernetes/kubernetes/pull/116045)).
|
||||
|
||||
## Getting involved
|
||||
|
||||
This feature is managed by [SIG Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling).
|
||||
Please join us and share your feedback. We look forward to hearing from you!
|
||||
|
||||
## How can I learn more?
|
||||
|
||||
- [The official document of the ContainerResource type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics)
|
||||
- [KEP-1610: Container Resource based Autoscaling](https://github.com/kubernetes/enhancements/tree/master/keps/sig-autoscaling/1610-container-resource-autoscaling)
|
||||
Loading…
Reference in New Issue