diff --git a/content/en/docs/tasks/administer-cluster/reconfigure-kubelet.md b/content/en/docs/tasks/administer-cluster/reconfigure-kubelet.md index e8ab24a52f..83bb8f3379 100644 --- a/content/en/docs/tasks/administer-cluster/reconfigure-kubelet.md +++ b/content/en/docs/tasks/administer-cluster/reconfigure-kubelet.md @@ -4,17 +4,20 @@ reviewers: - dawnchen title: Reconfigure a Node's Kubelet in a Live Cluster content_template: templates/task +min-kubernetes-server-version: v1.11 --- {{% capture overview %}} {{< feature-state for_k8s_version="v1.11" state="beta" >}} [Dynamic Kubelet Configuration](https://github.com/kubernetes/enhancements/issues/281) -allows you to change the configuration of each Kubelet in a live Kubernetes -cluster by deploying a ConfigMap and configuring each Node to use it. +allows you to change the configuration of each +{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} in a running Kubernetes cluster, +by deploying a {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}} and configuring +each {{< glossary_tooltip term_id="node" >}} to use it. {{< warning >}} -All Kubelet configuration parameters can be changed dynamically, +All kubelet configuration parameters can be changed dynamically, but this is unsafe for some parameters. Before deciding to change a parameter dynamically, you need a strong understanding of how that change will affect your cluster's behavior. Always carefully test configuration changes on a small set @@ -25,38 +28,49 @@ fields is available in the inline `KubeletConfiguration` {{% /capture %}} {{% capture prerequisites %}} -- Kubernetes v1.11 or higher on both the Master and the Nodes -- kubectl v1.11 or higher, configured to communicate with the cluster -- The Kubelet's `--dynamic-config-dir` flag must be set to a writable - directory on the Node. +You need to have a Kubernetes cluster. +You also need kubectl v1.11 or higher, configured to communicate with your cluster. +{{< version-check >}} +Your cluster API server version (eg v1.12) must be no more than one minor +version away from the version of kubectl that you are using. For example, +if your cluster is running v1.16 then you can use kubectl v1.15, v1.16 +or v1.17; other combinations +[aren't supported](/docs/setup/release/version-skew-policy/#kubectl). + +Some of the examples use the commandline tool +[jq](https://stedolan.github.io/jq/). You do not need `jq` to complete the task, +because there are manual alternatives. + +For each node that you're reconfiguring, you must set the kubelet +`--dynamic-config-dir` flag to a writable directory. {{% /capture %}} {{% capture steps %}} -## Reconfiguring the Kubelet on a Live Node in your Cluster +## Reconfiguring the kubelet on a running node in your cluster -### Basic Workflow Overview +### Basic workflow overview -The basic workflow for configuring a Kubelet in a live cluster is as follows: +The basic workflow for configuring a kubelet in a live cluster is as follows: 1. Write a YAML or JSON configuration file containing the -Kubelet's configuration. +kubelet's configuration. 2. Wrap this file in a ConfigMap and save it to the Kubernetes control plane. -3. Update the Kubelet's corresponding Node object to use this ConfigMap. +3. Update the kubelet's corresponding Node object to use this ConfigMap. -Each Kubelet watches a configuration reference on its respective Node object. -When this reference changes, the Kubelet downloads the new configuration, +Each kubelet watches a configuration reference on its respective Node object. +When this reference changes, the kubelet downloads the new configuration, updates a local reference to refer to the file, and exits. For the feature to work correctly, you must be running an OS-level service -manager (such as systemd), which will restart the Kubelet if it exits. When the -Kubelet is restarted, it will begin using the new configuration. +manager (such as systemd), which will restart the kubelet if it exits. When the +kubelet is restarted, it will begin using the new configuration. The new configuration completely overrides configuration provided by `--config`, and is overridden by command-line flags. Unspecified values in the new configuration will receive default values appropriate to the configuration version (e.g. `kubelet.config.k8s.io/v1beta1`), unless overridden by flags. -The status of the Node's Kubelet configuration is reported via +The status of the Node's kubelet configuration is reported via `Node.Spec.Status.Config`. Once you have updated a Node to use the new ConfigMap, you can observe this status to confirm that the Node is using the intended configuration. @@ -70,7 +84,7 @@ mind that it is also valid for multiple Nodes to consume the same ConfigMap. {{< warning >}} While it is *possible* to change the configuration by -updating the ConfigMap in-place, this causes all Kubelets configured with +updating the ConfigMap in-place, this causes all kubelets configured with that ConfigMap to update simultaneously. It is much safer to treat ConfigMaps as immutable by convention, aided by `kubectl`'s `--append-hash` option, and incrementally roll out updates to `Node.Spec.ConfigSource`. @@ -91,23 +105,35 @@ and debug issues. The compromise, however, is that you must start with knowledge of the existing configuration to ensure that you only change the fields you intend to change. -Ideally, the Kubelet would be bootstrapped from a file on disk -and you could edit this file (which could also be version-controlled), -to create the first Kubelet ConfigMap -(see [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)), -Currently, the Kubelet is bootstrapped with **a combination of this file and command-line flags** -that can override the configuration in the file. -As a workaround, you can generate a config file containing a Node's current -configuration by accessing the Kubelet server's `configz` endpoint via the -kubectl proxy. This endpoint, in its current implementation, is intended to be -used only as a debugging aid. Do not rely on the behavior of this endpoint for -production scenarios. The examples below use the `jq` command to streamline -working with JSON. To follow the tasks as written, you need to have `jq` -installed, but you can adapt the tasks if you prefer to extract the -`kubeletconfig` subobject manually. +The kubelet loads settings from its configuration file, but you can set command +line flags to override the configuration in the file. This means that if you +only know the contents of the configuration file, and you don't know the +command line overrides, then you do not know the running configuration either. + +Because you need to know the running configuration in order to override it, +you can fetch the running configuration from the kubelet. You can generate a +config file containing a Node's current configuration by accessing the kubelet's +`configz` endpoint, through `kubectl proxy`. The next section explains how to +do this. + +{{< caution >}} +The kubelet's `configz` endpoint is there to help with debugging, and is not +a stable part of kubelet behavior. +Do not rely on the behavior of this endpoint for production scenarios or for +use with automated tools. +{{< /caution >}} + +For more information on configuring the kubelet via a configuration file, see +[Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)). #### Generate the configuration file +{{< note >}} +The steps below use the `jq` command to streamline working with JSON. +To follow the tasks as written, you need to have `jq` installed. You can +adapt the steps if you prefer to extract the `kubeletconfig` subobject manually. +{{< /note >}} + 1. Choose a Node to reconfigure. In this example, the name of this Node is referred to as `NODE_NAME`. 2. Start the kubectl proxy in the background using the following command: @@ -122,20 +148,22 @@ installed, but you can adapt the tasks if you prefer to extract the For example: `${NODE_NAME}` will be rewritten as `$\{NODE_NAME\}` during the paste. You must remove the backslashes before running the command, or the command will fail. + ```bash NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME} ``` {{< note >}} You need to manually add the `kind` and `apiVersion` to the downloaded -object, because they are not reported by the `configz` endpoint. +object, because those fields are not reported by the `configz` endpoint. {{< /note >}} #### Edit the configuration file Using a text editor, change one of the parameters in the file generated by the previous procedure. For example, you -might edit the QPS parameter `eventRecordQPS`. +might edit the parameter `eventRecordQPS`, that controls +rate limiting for event recording. #### Push the configuration file to the control plane @@ -162,12 +190,12 @@ data: {...} ``` -The ConfigMap is created in the `kube-system` namespace because this -ConfigMap configures a Kubelet, which is a Kubernetes system component. +You created that ConfigMap inside the `kube-system` namespace because the kubelet +is a Kubernetes system component. The `--append-hash` option appends a short checksum of the ConfigMap contents to the name. This is convenient for an edit-then-push workflow, because it -automatically, yet deterministically, generates new names for new ConfigMaps. +automatically, yet deterministically, generates new names for new resources. The name that includes this generated hash is referred to as `CONFIG_MAP_NAME` in the following examples. @@ -185,13 +213,13 @@ In your text editor, add the following YAML under `spec`: ```yaml configSource: configMap: - name: CONFIG_MAP_NAME + name: CONFIG_MAP_NAME # replace CONFIG_MAP_NAME with the name of the ConfigMap namespace: kube-system kubeletConfigKey: kubelet ``` You must specify all three of `name`, `namespace`, and `kubeletConfigKey`. -The `kubeletConfigKey` parameter shows the Kubelet which key of the ConfigMap +The `kubeletConfigKey` parameter shows the kubelet which key of the ConfigMap contains its config. #### Observe that the Node begins using the new configuration @@ -200,16 +228,16 @@ Retrieve the Node using the `kubectl get node ${NODE_NAME} -o yaml` command and `Node.Status.Config`. The config sources corresponding to the `active`, `assigned`, and `lastKnownGood` configurations are reported in the status. -- The `active` configuration is the version the Kubelet is currently running with. -- The `assigned` configuration is the latest version the Kubelet has resolved based on +- The `active` configuration is the version the kubelet is currently running with. +- The `assigned` configuration is the latest version the kubelet has resolved based on `Node.Spec.ConfigSource`. - The `lastKnownGood` configuration is the version the - Kubelet will fall back to if an invalid config is assigned in `Node.Spec.ConfigSource`. + kubelet will fall back to if an invalid config is assigned in `Node.Spec.ConfigSource`. The`lastKnownGood` configuration might not be present if it is set to its default value, the local config deployed with the node. The status will update `lastKnownGood` to -match a valid `assigned` config after the Kubelet becomes comfortable with the config. -The details of how the Kubelet determines a config should become the `lastKnownGood` are +match a valid `assigned` config after the kubelet becomes comfortable with the config. +The details of how the kubelet determines a config should become the `lastKnownGood` are not guaranteed by the API, but is currently implemented as a 10-minute grace period. You can use the following command (using `jq`) to filter down @@ -254,16 +282,19 @@ The following is an example response: ``` -If an error occurs, the Kubelet reports it in the `Node.Status.Config.Error` +(if you do not have `jq`, you can look at the whole response and find `Node.Status.Config` +by eye). + +If an error occurs, the kubelet reports it in the `Node.Status.Config.Error` structure. Possible errors are listed in [Understanding Node.Status.Config.Error messages](#understanding-node-status-config-error-messages). -You can search for the identical text in the Kubelet log for additional details +You can search for the identical text in the kubelet log for additional details and context about the error. #### Make more changes Follow the workflow above to make more changes and push them again. Each time -you push a ConfigMap with new contents, the --append-hash kubectl option creates +you push a ConfigMap with new contents, the `--append-hash` kubectl option creates the ConfigMap with a new name. The safest rollout strategy is to first create a new ConfigMap, and then update the Node to use the new ConfigMap. @@ -283,7 +314,7 @@ error is reported. {{% /capture %}} {{% capture discussion %}} -## Kubectl Patch Example +## `kubectl patch` example You can change a Node's configSource using several different mechanisms. This example uses `kubectl patch`: @@ -292,25 +323,25 @@ This example uses `kubectl patch`: kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}" ``` -## Understanding how the Kubelet checkpoints config +## Understanding how the kubelet checkpoints config -When a new config is assigned to the Node, the Kubelet downloads and unpacks the -config payload as a set of files on the local disk. The Kubelet also records metadata +When a new config is assigned to the Node, the kubelet downloads and unpacks the +config payload as a set of files on the local disk. The kubelet also records metadata that locally tracks the assigned and last-known-good config sources, so that the -Kubelet knows which config to use across restarts, even if the API server becomes -unavailable. After checkpointing a config and the relevant metadata, the Kubelet -exits if it detects that the assigned config has changed. When the Kubelet is +kubelet knows which config to use across restarts, even if the API server becomes +unavailable. After checkpointing a config and the relevant metadata, the kubelet +exits if it detects that the assigned config has changed. When the kubelet is restarted by the OS-level service manager (such as `systemd`), it reads the new metadata and uses the new config. The recorded metadata is fully resolved, meaning that it contains all necessary information to choose a specific config version - typically a `UID` and `ResourceVersion`. This is in contrast to `Node.Spec.ConfigSource`, where the intended config is declared -via the idempotent `namespace/name` that identifies the target ConfigMap; the Kubelet +via the idempotent `namespace/name` that identifies the target ConfigMap; the kubelet tries to use the latest version of this ConfigMap. -When you are debugging problems on a node, you can inspect the Kubelet's config -metadata and checkpoints. The structure of the Kubelet's checkpointing directory is: +When you are debugging problems on a node, you can inspect the kubelet's config +metadata and checkpoints. The structure of the kubelet's checkpointing directory is: ```none - --dynamic-config-dir (root for managing dynamic config) @@ -334,13 +365,18 @@ in the Kubelet log for additional details and context about the error. Error Message | Possible Causes :-------------| :-------------- -failed to load config, see Kubelet log for details | The Kubelet likely could not parse the downloaded config payload, or encountered a filesystem error attempting to load the payload from disk. -failed to validate config, see Kubelet log for details | The configuration in the payload, combined with any command-line flag overrides, and the sum of feature gates from flags, the config file, and the remote payload, was determined to be invalid by the Kubelet. -invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil | Since Node.Spec.ConfigSource is validated by the API server to contain at least one non-nil subfield, this likely means that the Kubelet is older than the API server and does not recognize a newer source type. -failed to sync: failed to download config, see Kubelet log for details | The Kubelet could not download the config. It is possible that Node.Spec.ConfigSource could not be resolved to a concrete API object, or that network errors disrupted the download attempt. The Kubelet will retry the download when in this error state. -failed to sync: internal failure, see Kubelet log for details | The Kubelet encountered some internal problem and failed to update its config as a result. Examples include filesystem errors and reading objects from the internal informer cache. -internal failure, see Kubelet log for details | The Kubelet encountered some internal problem while manipulating config, outside of the configuration sync loop. +failed to load config, see Kubelet log for details | The kubelet likely could not parse the downloaded config payload, or encountered a filesystem error attempting to load the payload from disk. +failed to validate config, see Kubelet log for details | The configuration in the payload, combined with any command-line flag overrides, and the sum of feature gates from flags, the config file, and the remote payload, was determined to be invalid by the kubelet. +invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil | Since Node.Spec.ConfigSource is validated by the API server to contain at least one non-nil subfield, this likely means that the kubelet is older than the API server and does not recognize a newer source type. +failed to sync: failed to download config, see Kubelet log for details | The kubelet could not download the config. It is possible that Node.Spec.ConfigSource could not be resolved to a concrete API object, or that network errors disrupted the download attempt. The kubelet will retry the download when in this error state. +failed to sync: internal failure, see Kubelet log for details | The kubelet encountered some internal problem and failed to update its config as a result. Examples include filesystem errors and reading objects from the internal informer cache. +internal failure, see Kubelet log for details | The kubelet encountered some internal problem while manipulating config, outside of the configuration sync loop. -{{< /table >}} +{{< /table >}} {{% /capture %}} +{{% capture whatsnext %}} + - For more information on configuring the kubelet via a configuration file, see +[Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file). +- See the reference documentation for [`NodeConfigSource`](https://kubernetes.io/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#nodeconfigsource-v1-core) +{{% /capture %}}