Merge pull request #37768 from atiratree/pod-healthy-policy-for-pdbs

Add documentation for Unhealthy Pod Eviction Policy for PDBs

content/en/docs/reference/command-line-tools-reference/feature-gates.md

@@ -148,6 +148,7 @@ For a reference to old feature gates that are removed, please refer to
 | `OpenAPIEnums` | `true` | Beta | 1.24 | |
 | `OpenAPIV3` | `false` | Alpha | 1.23 | 1.23 |
 | `OpenAPIV3` | `true` | Beta | 1.24 | |
+| `PDBUnhealthyPodEvictionPolicy` | `false` | Alpha | 1.26 | |
 | `PodAndContainerStatsFromCRI` | `false` | Alpha | 1.23 | |
 | `PodDeletionCost` | `false` | Alpha | 1.21 | 1.21 |
 | `PodDeletionCost` | `true` | Beta | 1.22 | |
@@ -666,6 +667,9 @@ Each feature gate is designed for enabling/disabling a specific feature:
 - `OpenAPIEnums`: Enables populating "enum" fields of OpenAPI schemas in the
   spec returned from the API server.
 - `OpenAPIV3`: Enables the API server to publish OpenAPI v3.
+- `PDBUnhealthyPodEvictionPolicy`: Enables the `unhealthyPodEvictionPolicy` field of a `PodDisruptionBudget`. This specifies
+  when unhealthy pods should be considered for eviction. Please see [Unhealthy Pod Eviction Policy](/docs/tasks/run-application/configure-pdb/#unhealthy-pod-eviction-policy)
+  for more details.
 - `PodDeletionCost`: Enable the [Pod Deletion Cost](/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost)
    feature which allows users to influence ReplicaSet downscaling order.
 - `PodAffinityNamespaceSelector`: Enable the

content/en/docs/tasks/administer-cluster/safely-drain-node.md

@@ -88,10 +88,11 @@ respect the PodDisruptionBudget you specify.
 For example, if you have a StatefulSet with three replicas and have
 set a PodDisruptionBudget for that set specifying `minAvailable: 2`,
 `kubectl drain` only evicts a pod from the StatefulSet if all three
-replicas pods are ready; if then you issue multiple drain commands in
-parallel, Kubernetes respects the PodDisruptionBudget and ensure
-that only 1 (calculated as `replicas - minAvailable`) Pod is unavailable
-at any given time. Any drains that would cause the number of ready
+replicas pods are [healthy](/docs/tasks/run-application/configure-pdb/#healthiness-of-a-pod);
+if then you issue multiple drain commands in parallel,
+Kubernetes respects the PodDisruptionBudget and ensures that
+only 1 (calculated as `replicas - minAvailable`) Pod is unavailable
+at any given time. Any drains that would cause the number of [healthy](/docs/tasks/run-application/configure-pdb/#healthiness-of-a-pod)
 replicas to fall below the specified budget are blocked.
 
 ## The Eviction API {#eviction-api}

content/en/docs/tasks/run-application/configure-pdb.md

@@ -127,7 +127,7 @@ is the `scale` of the controller managing the pods being selected by the
 `PodDisruptionBudget`.
 
 Example 1: With a `minAvailable` of 5, evictions are allowed as long as they leave behind
-5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`.
+5 or more [healthy](#healthiness-of-a-pod) pods among those selected by the PodDisruptionBudget's `selector`.
 
 Example 2: With a `minAvailable` of 30%, evictions are allowed as long as at least 30%
 of the number of desired replicas are healthy. 
@@ -229,6 +229,51 @@ status:
   observedGeneration: 1
 ```
 
+### Healthiness of a Pod
+
+The current implementation considers healthy pods, as pods that have `.status.conditions` item with `type="Ready"` and `status="True"`.
+These pods are tracked via `.status.currentHealthy` field in the PDB status.
+
+## Unhealthy Pod Eviction Policy
+
+{{< feature-state for_k8s_version="v1.26" state="alpha" >}}
+
+{{< note >}}
+In order to use this behavior, you must enable the `PDBUnhealthyPodEvictionPolicy`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/).
+{{< /note >}}
+
+PodDisruptionBudget guarding an application ensures that `.status.currentHealthy` number of pods
+does not fall below the number specified in `.status.desiredHealthy` by disallowing eviction of healthy pods.
+By using `.spec.unhealthyPodEvictionPolicy`, you can also define the criteria when unhealthy pods 
+should be considered for eviction. The default behavior when no policy is specified corresponds 
+to the `IfHealthyBudget` policy.
+
+Policies:
+
+`IfHealthyBudget`
+: Running pods (`.status.phase="Running"`), but not yet healthy can be evicted only if the guarded application is not
+disrupted (`.status.currentHealthy` is at least equal to `.status.desiredHealthy`).
+
+: This policy ensures that running pods of an already disrupted application have the best chance to become healthy.
+This has negative implications for draining nodes, which can be blocked by misbehaving applications that are guarded by a PDB.
+More specifically applications with pods in `CrashLoopBackOff` state (due to a bug or misconfiguration),
+or pods that are just failing to report the `Ready` condition.
+
+`AlwaysAllow`
+: Running pods (`.status.phase="Running"`), but not yet healthy are considered disrupted and can be evicted
+regardless of whether the criteria in a PDB is met.
+
+: This means prospective running pods of a disrupted application might not get a chance to become healthy.
+By using this policy, cluster managers can easily evict misbehaving applications that are guarded by a PDB.
+More specifically applications with pods in `CrashLoopBackOff` state (due to a bug or misconfiguration),
+or pods that are just failing to report the `Ready` condition.
+
+{{< note >}}
+Pods in `Pending`, `Succeeded` or `Failed` phase are always considered for eviction.
+{{< /note >}}
+
 ## Arbitrary Controllers and Selectors
 
 You can skip this section if you only use PDBs with the built-in