203 lines
10 KiB
Markdown
203 lines
10 KiB
Markdown
---
|
|
reviewers:
|
|
- sig-cluster-lifecycle
|
|
title: Configuring each kubelet in your cluster using kubeadm
|
|
content_template: templates/concept
|
|
weight: 80
|
|
---
|
|
|
|
{{% capture overview %}}
|
|
|
|
{{< feature-state for_k8s_version="1.11" state="stable" >}}
|
|
|
|
The lifecycle of the kubeadm CLI tool is decoupled from the
|
|
[kubelet](/docs/reference/command-line-tools-reference/kubelet), which is a daemon that runs
|
|
on each node within the Kubernetes cluster. The kubeadm CLI tool is executed by the user when Kubernetes is
|
|
initialized or upgraded, whereas the kubelet is always running in the background.
|
|
|
|
Since the kubelet is a daemon, it needs to be maintained by some kind of a init
|
|
system or service manager. When the kubelet is installed using DEBs or RPMs,
|
|
systemd is configured to manage the kubelet. You can use a different service
|
|
manager instead, but you need to configure it manually.
|
|
|
|
Some kubelet configuration details need to be the same across all kubelets involved in the cluster, while
|
|
other configuration aspects need to be set on a per-kubelet basis, to accommodate the different
|
|
characteristics of a given machine, such as OS, storage, and networking. You can manage the configuration
|
|
of your kubelets manually, but [kubeadm now provides a `KubeletConfiguration` API type for managing your
|
|
kubelet configurations centrally](#configure-kubelets-using-kubeadm).
|
|
|
|
{{% /capture %}}
|
|
|
|
{{% capture body %}}
|
|
|
|
## Kubelet configuration patterns
|
|
|
|
The following sections describe patterns to kubelet configuration that are simplified by
|
|
using kubeadm, rather than managing the kubelet configuration for each Node manually.
|
|
|
|
### Propagating cluster-level configuration to each kubelet
|
|
|
|
You can provide the kubelet with default values to be used by `kubeadm init` and `kubeadm join`
|
|
commands. Interesting examples include using a different CRI runtime or setting the default subnet
|
|
used by services.
|
|
|
|
If you want your services to use the subnet `10.96.0.0/12` as the default for services, you can pass
|
|
the `--service-cidr` parameter to kubeadm:
|
|
|
|
```bash
|
|
kubeadm init --service-cidr 10.96.0.0/12
|
|
```
|
|
|
|
Virtual IPs for services are now allocated from this subnet. You also need to set the DNS address used
|
|
by the kubelet, using the `--cluster-dns` flag. This setting needs to be the same for every kubelet
|
|
on every manager and Node in the cluster. The kubelet provides a versioned, structured API object
|
|
that can configure most parameters in the kubelet and push out this configuration to each running
|
|
kubelet in the cluster. This object is called **the kubelet's ComponentConfig**.
|
|
The ComponentConfig allows the user to specify flags such as the cluster DNS IP addresses expressed as
|
|
a list of values to a camelCased key, illustrated by the following example:
|
|
|
|
```yaml
|
|
apiVersion: kubelet.config.k8s.io/v1beta1
|
|
kind: KubeletConfiguration
|
|
clusterDNS:
|
|
- 10.96.0.10
|
|
```
|
|
|
|
For more details on the ComponentConfig have a look at [this section](#configure-kubelets-using-kubeadm).
|
|
|
|
### Providing instance-specific configuration details
|
|
|
|
Some hosts require specific kubelet configurations, due to differences in hardware, operating system,
|
|
networking, or other host-specific parameters. The following list provides a few examples.
|
|
|
|
- The path to the DNS resolution file, as specified by the `--resolv-conf` kubelet
|
|
configuration flag, may differ among operating systems, or depending on whether you are using
|
|
`systemd-resolved`. If this path is wrong, DNS resolution will fail on the Node whose kubelet
|
|
is configured incorrectly.
|
|
|
|
- The Node API object `.metadata.name` is set to the machine's hostname by default,
|
|
unless you are using a cloud provider. You can use the `--hostname-override` flag to override the
|
|
default behavior if you need to specify a Node name different from the machine's hostname.
|
|
|
|
- Currently, the kubelet cannot automatically detects the cgroup driver used by the CRI runtime,
|
|
but the value of `--cgroup-driver` must match the cgroup driver used by the CRI runtime to ensure
|
|
the health of the kubelet.
|
|
|
|
- Depending on the CRI runtime your cluster uses, you may need to specify different flags to the kubelet.
|
|
For instance, when using Docker, you need to specify flags such as `--network-plugin=cni`, but if you
|
|
are using an external runtime, you need to specify `--container-runtime=remote` and specify the CRI
|
|
endpoint using the `--container-runtime-path-endpoint=<path>`.
|
|
|
|
You can specify these flags by configuring an individual kubelet's configuration in your service manager,
|
|
such as systemd.
|
|
|
|
## Configure kubelets using kubeadm
|
|
|
|
It is possible to configure the kubelet that kubeadm will start if a custom `KubeletConfiguration`
|
|
API object is passed with a configuration file like so `kubeadm ... --config some-config-file.yaml`.
|
|
|
|
By calling `kubeadm config print-default --api-objects KubeletConfiguration` you can
|
|
see all the default values for this structure.
|
|
|
|
Also have a look at the [API reference for the
|
|
kubelet ComponentConfig](https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/config#KubeletConfiguration)
|
|
for more information on the individual fields.
|
|
|
|
### Workflow when using `kubeadm init`
|
|
|
|
When you call `kubeadm init`, the kubelet configuration is marshalled to disk
|
|
at `/var/lib/kubelet/config.yaml`, and also uploaded to a ConfigMap in the cluster. The ConfigMap
|
|
is named `kubelet-config-1.X`, where `.X` is the minor version of the Kubernetes version you are
|
|
initializing. A kubelet configuration file is also written to `/etc/kubernetes/kubelet.conf` with the
|
|
baseline cluster-wide configuration for all kubelets in the cluster. This configuration file
|
|
points to the client certificates that allow the kubelet to communicate with the API server. This
|
|
addresses the need to
|
|
[propagate cluster-level configuration to each kubelet](#propagating-cluster-level-configuration-to-each-kubelet).
|
|
|
|
To address the second pattern of
|
|
[providing instance-specific configuration details](#providing-instance-specific-configuration-details),
|
|
kubeadm writes an environment file to `/var/lib/kubelet/kubeadm-flags.env`, which contains a list of
|
|
flags to pass to the kubelet when it starts. The flags are presented in the file like this:
|
|
|
|
```bash
|
|
KUBELET_KUBEADM_ARGS="--flag1=value1 --flag2=value2 ..."
|
|
```
|
|
|
|
In addition to the flags used when starting the kubelet, the file also contains dynamic
|
|
parameters such as the cgroup driver and whether to use a different CRI runtime socket
|
|
(`--cri-socket`).
|
|
|
|
After marshalling these two files to disk, kubeadm attempts to run the following two
|
|
commands, if you are using systemd:
|
|
|
|
```bash
|
|
systemctl daemon-reload && systemctl restart kubelet
|
|
```
|
|
|
|
If the reload and restart are successful, the normal `kubeadm init` workflow continues.
|
|
|
|
### Workflow when using `kubeadm join`
|
|
|
|
When you run `kubeadm join`, kubeadm uses the Bootstrap Token credential perform
|
|
a TLS bootstrap, which fetches the credential needed to download the
|
|
`kubelet-config-1.X` ConfigMap and writes it to `/var/lib/kubelet/config.yaml`. The dynamic
|
|
environment file is generated in exactly the same way as `kubeadm init`.
|
|
|
|
Next, `kubeadm` runs the following two commands to load the new configuration into the kubelet:
|
|
|
|
```bash
|
|
systemctl daemon-reload && systemctl restart kubelet
|
|
```
|
|
|
|
After the kubelet loads the new configuration, kubeadm writes the
|
|
`/etc/kubernetes/bootstrap-kubelet.conf` KubeConfig file, which contains a CA certificate and Bootstrap
|
|
Token. These are used by the kubelet to perform the TLS Bootstrap and obtain a unique
|
|
credential, which is stored in `/etc/kubernetes/kubelet.conf`. When this file is written, the kubelet
|
|
has finished performing the TLS Bootstrap.
|
|
|
|
## The kubelet drop-in file for systemd
|
|
|
|
The configuration file installed by the kubeadm DEB or RPM package is written to
|
|
`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` and is used by systemd.
|
|
|
|
```none
|
|
[Service]
|
|
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
|
|
--kubeconfig=/etc/kubernetes/kubelet.conf"
|
|
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
|
|
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating
|
|
the KUBELET_KUBEADM_ARGS variable dynamically
|
|
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
|
|
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
|
|
#the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
|
|
# KUBELET_EXTRA_ARGS should be sourced from this file.
|
|
EnvironmentFile=-/etc/default/kubelet
|
|
ExecStart=
|
|
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
|
|
```
|
|
|
|
This file specifies the default locations for all of the files managed by kubeadm for the kubelet.
|
|
|
|
- The KubeConfig file to use for the TLS Bootstrap is `/etc/kubernetes/bootstrap-kubelet.conf`,
|
|
but it is only used if `/etc/kubernetes/kubelet.conf` does not exist.
|
|
- The KubeConfig file with the unique kubelet identity is `/etc/kubernetes/kubelet.conf`.
|
|
- The file containing the kubelet's ComponentConfig is `/var/lib/kubelet/config.yaml`.
|
|
- The dynamic environment file that contains `KUBELET_KUBEADM_ARGS` is sourced from `/var/lib/kubelet/kubeadm-flags.env`.
|
|
- The file that can contain user-specified flag overrides with `KUBELET_EXTRA_ARGS` is sourced from
|
|
`/etc/default/kubelet` (for DEBs), or `/etc/systconfig/kubelet` (for RPMs). `KUBELET_EXTRA_ARGS`
|
|
is last in the flag chain and has the highest priority in the event of conflicting settings.
|
|
|
|
## Kubernetes binaries and package contents
|
|
|
|
The DEB and RPM packages shipped with the Kubernetes releases are:
|
|
|
|
| Package name | Description |
|
|
|--------------|-------------|
|
|
| `kubeadm` | Installs the `/usr/bin/kubeadm` CLI tool and [The kubelet drop-in file(#the-kubelet-drop-in-file-for-systemd) for the kubelet. |
|
|
| `kubelet` | Installs the `/usr/bin/kubelet` binary. |
|
|
| `kubectl` | Installs the `/usr/bin/kubectl` binary. |
|
|
| `kubernetes-cni` | Installs the official CNI binaries into the `/opt/cni/bin` directory. |
|
|
| `cri-tools` | Installs the `/usr/bin/crictl` binary from [https://github.com/kubernetes-incubator/cri-tools](https://github.com/kubernetes-incubator/cri-tools). |
|
|
|
|
{{% /capture %}}
|