kubernetes
/
website
mirror of https://github.com/kubernetes/website.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update content/en/blog/_posts/2023-08-04-kubernetes-1.28-blog.md 

Co-authored-by: Tim Bannister <tim@scalefactory.com>
pull/42381/head
Brad McCoy 2023-08-14 18:21:47 +10:00 committed by GitHub
parent 6bd5ff3990
commit e7cb7e5647
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 14 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
content/en/blog/_posts/2023-08-04-kubernetes-1.28-blog.md
View File

@ -41,7 +41,20 @@ shutdown might not get handled well; that could be because:
- because of a node-level configuration error
(such as not setting appropriate values for `shutdownGracePeriod` and `shutdownGracePeriodCriticalPods`).
When a node is shutdown but not detected by Kubelet's Node Shutdown Manager, the pods that are part of a StatefulSet will be stuck in terminating status on the shutdown node and cannot move to a new running node. This is because Kubelet on the shutdown node is not available to delete the pods so the StatefulSet cannot create a new pod with the same name. If there are volumes used by the pods, the VolumeAttachments will not be deleted from the original shutdown node so the volumes used by these pods cannot be attached to a new running node. As a result, the application running on the StatefulSet cannot function properly. If the original shutdown node comes up, the pods will be deleted by Kubelet and new pods will be created on a different running node. If the original shutdown node does not come up, these pods will be stuck in terminating status on the shutdown node forever.
When a node is shutdown or fails, and that shutdown was not detected by the kubelet, the pods that are part
of a StatefulSet will be stuck in terminating status on the shutdown node. If the stopped node restarts, the
kubelet on that node can clean up (`DELETE`) the Pods that the Kubernetes API still sees as bound to that node.
However, if the node stays stopped - or if the kubelet isn't able to start after a reboot - then Kubernetes may
not be able to create replacement Pods. When the kubelet on the shut-down node is not available to delete
the old pods, an associated StatefulSet cannot create a new pod (which would have the same name).
There's also a problem with storage. If there are volumes used by the pods, existing VolumeAttachments will
not be disassociated from the original - and now shut down - node so the PersistentVolumes used by these
pods cannot be attached to a different, healthy node. As a result, an application running on an
affected StatefulSet may not be able to function properly. If the original, shut down node does come up, the
there pods will be deleted by its kubelet and new pods can be created on a different running node.
If the original node does not come up (common with an [immutable infrastructure](https://glossary.cncf.io/immutable-infrastructure/) design),
those pods would be stuck in a `Terminating` status on the shut-down node forever.
This handles node shutdown cases that are not detected by Kubelet's Node Shutdown Manager. The pods will be forcefully deleted in this case, triggering the deletion of the VolumeAttachments, and new pods will be created on a new running node so that application can continue to function.