diff --git a/_data/tasks.yml b/_data/tasks.yml index bfc5c9426f2..6fbe0809e58 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -54,3 +54,7 @@ toc: - docs/tasks/manage-stateful-set/deleting-a-statefulset.md - docs/tasks/manage-stateful-set/debugging-a-statefulset.md - docs/tasks/manage-stateful-set/delete-pods.md + +- title: Managing Cluster Daemons + section: + - docs/tasks/manage-daemon/update-daemon-set.md diff --git a/docs/admin/daemons.md b/docs/admin/daemons.md index 4682a62b711..d90183ae511 100644 --- a/docs/admin/daemons.md +++ b/docs/admin/daemons.md @@ -121,7 +121,7 @@ the new DaemonSet with the different template will recognize all the existing po matching labels. It will not modify or delete them despite a mismatch in the pod template. You will need to force new pod creation by deleting the pod or deleting the node. -You cannot update a DaemonSet. +Starting from Kubernetes version 1.6, you can [perform a rolling update](/docs/tasks/manage-daemon/update-daemon-set/) on a DaemonSet. Support for updating DaemonSets and controlled updating of nodes is planned. diff --git a/docs/tasks/index.md b/docs/tasks/index.md index 4572ca077b9..b29ad1764e2 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -63,6 +63,9 @@ single thing, typically by giving a short sequence of steps. * [Debugging a StatefulSet](/docs/tasks/manage-stateful-set/debugging-a-statefulset/) * [Force Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/) +#### Managing Cluster Daemons +* [Performing a Rolling Update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/) + ### What's next If you would like to write a task page, see diff --git a/docs/tasks/manage-daemon/update-daemon-set.md b/docs/tasks/manage-daemon/update-daemon-set.md new file mode 100644 index 00000000000..81de12d46d1 --- /dev/null +++ b/docs/tasks/manage-daemon/update-daemon-set.md @@ -0,0 +1,232 @@ +--- +assignees: +- janetkuo +title: Performing a Rolling Update on a DaemonSet +--- + +{% capture overview %} + +This page shows how to perform a rolling update on a DaemonSet. + +{% endcapture %} + + +{% capture prerequisites %} + +* The DaemonSet rolling update feature is only supported in Kubernetes version 1.6 or later. + +{% endcapture %} + + +{% capture steps %} + +## DaemonSet Update Strategy + +DaemonSet has two update strategy types : + +* OnDelete: This is the default update strategy for backward-compatibility. With + `OnDelete` update strategy, after you update a DaemonSet template, new + DaemonSet pods will *only* be created when you manually delete old DaemonSet + pods. This is the same behavior of DaemonSet in Kubernetes version 1.5 or + before. +* RollingUpdate: With `RollingUpdate` update strategy, after you update a + DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods + will be created automatically, in a controlled fashion. + +## Limitations + +* DaemonSet rollout history is not supported yet. +* DaemonSet rollback is not directly supported in `kubectl` yet. You can rollback + by updating DaemonSet template to match the previous version. + +## Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before + +If you try to rolling update a DaemonSet that was created from Kubernetes +version 1.5 or before, a rollout will be triggered when you *first* change the +DaemonSet update strategy to `RollingUpdate`, even when DaemonSet template isn't +modified. All existing DaemonSet pods will be restarted. + +To avoid this restart, first find the DaemonSet's current +`.spec.templateGeneration`: + +```shell{% raw %} +kubectl get ds/ -o go-template='{{.spec.templateGeneration}}{{"\n"}}' +{% endraw %}``` + +The output should be a number *N*. If the output shows ``, N = 0. + +Then, simply label the existing DaemonSet pods with `pod-template-generation=` +before changing DaemonSet `.spec.updateStrategy` to `RollingUpdate`: + +```shell +# Replace N with DaemonSet `.spec.templateGeneration` +# Only run this on pods created from current DaemonSet template +kubectl label pods -l = pod-template-generation= +``` + +This tells the DaemonSet that the labeled DaemonSet pods are created from current +DaemonSet template. Therefore, you should only run this command to existing DaemonSet +pods that are generated from current DaemonSet template. + +Note that you only need to do this when you first change the update strategy of +a DaemonSet created from Kubernetes version 1.5 or before to `RollingUpdate`, +but don't want to update its template and start a new rollout yet. + +## Setting DaemonSet update strategy for rolling update + +To enable the rolling update feature of a DaemonSet, you must set its +`.spec.updateStrategy.type` to `RollingUpdate`. + +You may want to set `.spec.updateStrategy.rollingUpdate.maxUnavailable` (default +to 1) and `.spec.minReadySeconds` (default to 0) as well. + + +### Step 1: Checking DaemonSet `RollingUpdate` update strategy + +First, check the update strategy of your DaemonSet, and make sure it's set to +RollingUpdate: + +```shell{% raw %} +kubectl get ds/ -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'{% endraw %} +``` + +If you haven't created the DaemonSet in the system, check your DaemonSet +manifest with the following command instead: + +```shell{% raw %} +kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}' +{% endraw %}``` + +The output from both commands should be: + +```shell +RollingUpdate +``` + +If the output isn't `RollingUpdate`, go back and modify the DaemonSet object or +manifest accordingly. + +### Step 2: Creating a DaemonSet with `RollingUpdate` update strategy + +If you have already created the DaemonSet, you may skip this step and jump to +step 3. + +After verifying the update strategy of the DaemonSet manifest, create the DaemonSet: + +```shell +kubectl create -f ds.yaml +``` + +Alternatively, use `kubectl apply` to create the same DaemonSet if you plan to +update the DaemonSet with `kubectl apply`. + +```shell +kubectl apply -f ds.yaml +``` + +### Step 3: Updating a DaemonSet template + +Any updates to a `RollingUpdate` DaemonSet `.spec.template` will trigger a rolling +update. This can be done with several different `kubectl` commands. + +#### Declarative commands + +If you update DaemonSets using +[configuration files](/docs/tutorials/object-management-kubectl/declarative-object-management-configuration/), +use `kubectl apply`: + +```shell +kubectl apply -f ds-v2.yaml +``` + +#### Imperative commands + +If you update DaemonSets using +[imperative commands](/docs/tutorials/object-management-kubectl/imperative-object-management-command/), +use `kubectl edit` or `kubectl patch`: + +```shell +kubectl edit ds/ +``` + +```shell +kubectl patch ds/ -p= +``` + +##### Updating only the container image + +If you just need to update the container image in the DaemonSet template, i.e. +`.spec.template.spec.containers[*].image`, use `kubectl set image`: + +```shell +kubectl set image ds/ = +``` + +### Step 4: Watching the rolling update status + +Finally, watch the rollout status of the latest DaemonSet rolling update: + +```shell +kubectl rollout status ds/ +``` + +When the rollout is complete, the output is similar to this: + +```shell +daemon set "" successfully rolled out +``` + +## Troubleshooting + +### DaemonSet rolling update is stuck + +Sometimes, a DaemonSet rolling update may be stuck. Here are some possible +causes: + +#### Some nodes run out of resources + +The rollout is stuck because new DaemonSet pods can't be scheduled on at least one +node. This is possible when the node is +[running out of resources](/docs/concepts/cluster-administration/out-of-resource/). + +When this happens, find the nodes that don't have the DaemonSet pods scheduled on +by comparing the output of `kubectl get nodes` and the output of: + +```shell +kubectl get pods -l = -o wide +``` + +Once you've found those nodes, delete some non-DaemonSet pods from the node to +make room for new DaemonSet pods. Note that this will cause service disruption +if the deleted pods are not controlled by any controllers, or if the pods aren't +replicated. This doesn't respect +[PodDisruptionBudget](/docs/tasks/configure-pod-container/configure-pod-disruption-budget/) +either. + +#### Broken rollout + +If the recent DaemonSet template update is broken, for example, the container is +crash looping, or the container image doesn't exist (often due to a typo), +DaemonSet rollout won't progress. + +To fix this, just update the DaemonSet template again. New rollout won't be +blocked by previous unhealthy rollouts. + +#### Clock skew + +If `.spec.minReadySeconds` is specified in the DaemonSet, clock skew between +master and nodes will make DaemonSet unable to detect the right rollout +progress. + + +{% endcapture %} + + +{% capture whatsnext %} + +*TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"* + +{% endcapture %} + + +{% include templates/task.md %}