From 6c3808ec10413626acda9f6c6f803ad7e80cc085 Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Tue, 22 Oct 2024 12:25:58 -0700 Subject: [PATCH 1/2] In-Place Pod Resize Beta --- .../en/docs/concepts/workloads/autoscaling.md | 2 +- .../in-place-pod-vertical-scaling.md | 3 + .../en/docs/reference/node/kubelet-files.md | 2 +- .../resize-container-resources.md | 61 +++++++++++-------- 4 files changed, 42 insertions(+), 26 deletions(-) diff --git a/content/en/docs/concepts/workloads/autoscaling.md b/content/en/docs/concepts/workloads/autoscaling.md index ff154d71599..f2a5f1fbe07 100644 --- a/content/en/docs/concepts/workloads/autoscaling.md +++ b/content/en/docs/concepts/workloads/autoscaling.md @@ -79,7 +79,7 @@ Mode | Description #### Requirements for in-place resizing -{{< feature-state for_k8s_version="v1.27" state="alpha" >}} +{{< feature-state feature_gate_name="InPlacePodVerticalScaling" >}} Resizing a workload in-place **without** restarting the {{< glossary_tooltip text="Pods" term_id="pod" >}} or its {{< glossary_tooltip text="Containers" term_id="container" >}} requires Kubernetes version 1.27 or later. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md index 87e402385f6..353d3ea3a22 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md @@ -9,5 +9,8 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.27" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- Enables in-place Pod vertical scaling. diff --git a/content/en/docs/reference/node/kubelet-files.md b/content/en/docs/reference/node/kubelet-files.md index d50b3952947..321ffb65483 100644 --- a/content/en/docs/reference/node/kubelet-files.md +++ b/content/en/docs/reference/node/kubelet-files.md @@ -94,7 +94,7 @@ The name of a checkpoint file is `kubelet_internal_checkpoint` for [Device Manag If your cluster has [in-place Pod vertical scaling](/docs/concepts/workloads/autoscaling/#in-place-resizing) enabled ([feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -name `InPlacePodVerticalScaling`), then the kubelet stores a local record of Pod status. +name `InPlacePodVerticalScaling`), then the kubelet stores a local record of allocated Pod resources. The file name is `pod_status_manager_state` within the kubelet base directory (`/var/lib/kubelet` by default on Linux; configurable using `--root-dir`). diff --git a/content/en/docs/tasks/configure-pod-container/resize-container-resources.md b/content/en/docs/tasks/configure-pod-container/resize-container-resources.md index 8390605fd26..d66befd14af 100644 --- a/content/en/docs/tasks/configure-pod-container/resize-container-resources.md +++ b/content/en/docs/tasks/configure-pod-container/resize-container-resources.md @@ -24,26 +24,33 @@ to be enabled. The alternative is to delete the Pod and let the [workload controller](/docs/concepts/workloads/controllers/) make a replacement Pod that has a different resource requirement. +A resize request is made through the pod `/resize` subresource, which takes the full updated pod for +an update request, or a patch on the pod object for a patch request. + For in-place resize of pod resources: -- Container's resource `requests` and `limits` are _mutable_ for CPU - and memory resources. -- `allocatedResources` field in `containerStatuses` of the Pod's status reflects - the resources allocated to the pod's containers. -- `resources` field in `containerStatuses` of the Pod's status reflects the - actual resource `requests` and `limits` that are configured on the running - containers as reported by the container runtime. -- `resize` field in the Pod's status shows the status of the last requested +- A container's resource `requests` and `limits` are _mutable_ for CPU + and memory resources. These fields represent the _desired_ resources for the container. +- The `resources` field in `containerStatuses` of the Pod's status reflects the resources + _allocated_ to the pod's containers. For running containers, this reflects the actual resource + `requests` and `limits` that are configured as reported by the container runtime. For non-running + containers, these are the resources allocated for the container when it starts. +- The `resize` field in the Pod's status shows the status of the last requested pending resize. It can have the following values: - - `Proposed`: This value indicates an acknowledgement of the requested resize - and that the request was validated and recorded. + - `Proposed`: This value indicates that a pod was resized, but the Kubelet has not yet processed + the resize. - `InProgress`: This value indicates that the node has accepted the resize request and is in the process of applying it to the pod's containers. - `Deferred`: This value means that the requested resize cannot be granted at this time, and the node will keep retrying. The resize may be granted when - other pods leave and free up node resources. + other pods are removed and free up node resources. - `Infeasible`: is a signal that the node cannot accommodate the requested resize. This can happen if the requested resize exceeds the maximum resources the node can ever allocate for a pod. + - `""`: An empty or unset value indicates that the last resize completed. This should only be the + case if the resources in the container spec match the resources in the container status. + +If a node has pods with an incomplete resize, the scheduler will compute the pod requests from the +maximum of a container's desired resource requests, and it's actual requests reported in the status. ## {{% heading "prerequisites" %}} @@ -107,6 +114,21 @@ have changed, the container will be restarted in order to resize its memory. +## Limitations + +In-place resize of pod resources currently has the following limitations: + +- Only CPU and memory resources can be changed. +- Pod QoS Class cannot change. This means that requests must continue to equal limits for Guaranteed + pods, Burstable pods cannot set requests and limits to be equal for both CPU & memory, and you + cannot add resource requirements to Best Effort pods. +- Init containers and Ephemeral Containers cannot be resized. +- Resource requests and limits cannot be removed once set. +- A container's memory limit may not be reduced below its usage. If a request puts a container in + this state, the resize status will remain in `InProgress` until the desired memory limit becomes + feasible. +- Windows pods cannot be resized. + ## Create a pod with resource requests and limits @@ -159,9 +181,6 @@ spec: name: qos-demo-ctr-5 ready: true ... - allocatedResources: - cpu: 700m - memory: 200Mi resources: limits: cpu: 700m @@ -190,7 +209,7 @@ resources, you cannot change the QoS class in which the Pod was created. Now, patch the Pod's Container with CPU requests & limits both set to `800m`: ```shell -kubectl -n qos-example patch pod qos-demo-5 --patch '{"spec":{"containers":[{"name":"qos-demo-ctr-5", "resources":{"requests":{"cpu":"800m"}, "limits":{"cpu":"800m"}}}]}}' +kubectl -n qos-example patch pod qos-demo-5 --subresource resize --patch '{"spec":{"containers":[{"name":"qos-demo-ctr-5", "resources":{"requests":{"cpu":"800m"}, "limits":{"cpu":"800m"}}}]}}' ``` Query the Pod's detailed information after the Pod has been patched. @@ -215,9 +234,6 @@ spec: ... containerStatuses: ... - allocatedResources: - cpu: 800m - memory: 200Mi resources: limits: cpu: 800m @@ -229,12 +245,9 @@ spec: started: true ``` -Observe that the `allocatedResources` values have been updated to reflect the new -desired CPU requests. This indicates that node was able to accommodate the -increased CPU resource needs. - -In the Container's status, updated CPU resource values shows that new CPU -resources have been applied. The Container's `restartCount` remains unchanged, +Observe that the `resources` in the `containerStatuses` have been updated to reflect the new desired +CPU requests. This indicates that node was able to accommodate the increased CPU resource needs, +and the new CPU resources have been applied. The Container's `restartCount` remains unchanged, indicating that container's CPU resources were resized without restarting the container. From 1a730fd6d09b520eeb516ce86d0a0f21936a4a76 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 26 Nov 2024 19:12:20 +0000 Subject: [PATCH 2/2] Drop the feature promotion --- .../feature-gates/in-place-pod-vertical-scaling.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md index 353d3ea3a22..87e402385f6 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/in-place-pod-vertical-scaling.md @@ -9,8 +9,5 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.27" - - stage: beta - defaultValue: true - fromVersion: "1.32" --- Enables in-place Pod vertical scaling.