kubeadm: update authored docs for 1.32

pull/48408/head
Lubomir I. Ivanov 2024-11-11 11:04:07 +02:00
parent ed06b8628b
commit bbdb8dd9f3
4 changed files with 210 additions and 48 deletions

View File

@ -35,7 +35,7 @@ The cluster that `kubeadm init` and `kubeadm join` set up should be:
- **User-friendly**: The user should not have to run anything more than a couple of commands:
- `kubeadm init`
- `export KUBECONFIG=/etc/kubernetes/admin.conf`
- `kubectl apply -f <network-of-choice.yaml>`
- `kubectl apply -f <network-plugin-of-choice.yaml>`
- `kubeadm join --token <token> <endpoint>:<port>`
- **Extendable**:
- It should _not_ favor any particular network provider. Configuring the cluster network is out-of-scope
@ -76,6 +76,23 @@ in a majority of cases, and the most intuitive location; other constant paths an
- `front-proxy-ca.crt`, `front-proxy-ca.key` for the front proxy certificate authority
- `front-proxy-client.crt`, `front-proxy-client.key` for the front proxy client
## The kubeadm configuration file format
Most kubeadm commands support a `--config` flag which allows passing a configuration file from
disk. The configuration file format follows the common Kubernetes API `apiVersion` / `kind` scheme,
but is considered a component configuration format. Several Kubernetes components, such as the kubelet,
also support file-based configuration.
Different kubeadm subcommands require a different `kind` of configuration file.
For example, `InitConfiguration` for `kubeadm init`, `JoinConfiguration` for `kubeadm join`, `UpgradeConfiguration` for `kubeadm upgrade` and `ResetConfiguration`
for `kubeadm reset`.
The command `kubeadm config migrate` can be used to migrate an older format configuration
file to a newer (current) configuration format. The kubeadm tool only supports migrating from
deprecated configuration formats to the current format.
See the [kubeadm configuration reference](/docs/reference/config-api/kubeadm-config.v1beta4/) page for more details.
## kubeadm init workflow internal design
The `kubeadm init` consists of a sequence of atomic work tasks to perform,
@ -110,7 +127,7 @@ The user can skip specific preflight checks or all of them with the `--ignore-pr
- [Error] if `/etc/kubernetes/manifest` folder already exists and it is not empty
- [Error] if swap is on
- [Error] if `ip`, `iptables`, `mount`, `nsenter` commands are not present in the command path
- [Warning] if `ebtables`, `ethtool`, `socat`, `tc`, `touch`, `crictl` commands are not present in the command path
- [Warning] if `ethtool`, `tc`, `touch` commands are not present in the command path
- [Warning] if extra arg flags for API server, controller manager, scheduler contains some invalid options
- [Warning] if connection to https://API.AdvertiseAddress:API.BindPort goes through proxy
- [Warning] if connection to services subnet goes through proxy (only first address checked)
@ -188,7 +205,7 @@ Please note that:
Kubeadm generates kubeconfig files with identities for control plane components:
- A kubeconfig file for the kubelet to use during TLS bootstrap -
/etc/kubernetes/bootstrap-kubelet.conf. Inside this file, there is a bootstrap-token or embedded
`/etc/kubernetes/bootstrap-kubelet.conf`. Inside this file, there is a bootstrap-token or embedded
client certificates for authenticating this node with the cluster.
This client certificate should:
@ -223,7 +240,15 @@ The `super-admin.conf` file must be stored in a safe location and should not be
See [RBAC user facing role bindings](/docs/reference/access-authn-authz/rbac/#user-facing-roles)
for additional information on RBAC and built-in ClusterRoles and groups.
Please note that:
You can run [`kubeadm kubeconfig user`](/docs/reference/setup-tools/kubeadm/kubeadm-kubeconfig/#cmd-kubeconfig-user)
to generate kubeconfig files for additional users.
{{< caution >}}
The generated configuration files include an embedded authentication key, and you should treat
them as confidential.
{{< /caution >}}
Also note that:
1. `ca.crt` certificate is embedded in all the kubeconfig files.
1. If a given kubeconfig file exists, and its content is evaluated as compliant with the above specs,
@ -253,7 +278,7 @@ Static Pod manifests share a set of common properties:
- Leader election is enabled for both the controller-manager and the scheduler
- Controller-manager and the scheduler will reference kubeconfig files with their respective, unique identities
- All static Pods get any extra flags specified by the user as described in
- All static Pods get any extra flags or patches that you specify, as described in
[passing custom arguments to control plane components](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/)
- All static Pods get any extra Volumes specified by the user (Host path)
@ -344,10 +369,6 @@ the users:
- `--allocate-node-cidrs=true`
- `--cluster-cidr` and `--node-cidr-mask-size` flags according to the given CIDR
- If a cloud provider is specified, the corresponding `--cloud-provider` is specified together
with the `--cloud-config` path if such configuration file exists (this is experimental, alpha
and will be removed in a future version)
Other flags that are set unconditionally are:
- `--controllers` enabling all the default controllers plus `BootstrapSigner` and `TokenCleaner`
@ -365,7 +386,7 @@ Other flags that are set unconditionally are:
#### Scheduler
The static Pod manifest for the scheduler is not affected by parameters provided by the users.
The static Pod manifest for the scheduler is not affected by parameters provided by the user.
### Generate static Pod manifest for local etcd
@ -389,12 +410,10 @@ Please note that:
### Wait for the control plane to come up
kubeadm waits (upto 4m0s) until `localhost:6443/healthz` (kube-apiserver liveness) returns `ok`.
However, in order to detect deadlock conditions, kubeadm fails fast if `localhost:10255/healthz`
(kubelet liveness) or `localhost:10255/healthz/syncloop` (kubelet readiness) don't return `ok`
within 40s and 60s respectively.
On control plane nodes, kubeadm waits up to 4 minutes for the control plane components
and the kubelet to be available. It does that by performing a health check on the respective
component `/healthz` or `/livez` endpoints.
kubeadm relies on the kubelet to pull the control plane images and run them properly as static Pods.
After the control plane is up, kubeadm completes the tasks described in following paragraphs.
### Save the kubeadm ClusterConfiguration in a ConfigMap for later reference
@ -518,9 +537,8 @@ deployed as a DaemonSet:
#### DNS
- The CoreDNS service is named `kube-dns`. This is done to prevent any interruption
in service when the user is switching the cluster DNS from kube-dns to CoreDNS
through the `--config` method described [here](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon).
- The CoreDNS service is named `kube-dns` for compatibility reasons with the legacy `kube-dns`
addon.
- A ServiceAccount for CoreDNS is created in the `kube-system` namespace.
@ -534,8 +552,8 @@ You can use CoreDNS with kubeadm even when the related Service is named `kube-dn
Similarly to `kubeadm init`, also `kubeadm join` internal workflow consists of a sequence of
atomic work tasks to perform.
This is split into discovery (having the Node trust the Kubernetes Master) and TLS bootstrap
(having the Kubernetes Master trust the Node).
This is split into discovery (having the Node trust the Kubernetes API Server) and TLS bootstrap
(having the Kubernetes API Server trust the Node).
see [Authenticating with Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/)
or the corresponding [design proposal](https://git.k8s.io/design-proposals-archive/cluster-lifecycle/bootstrap-discovery.md).
@ -545,12 +563,10 @@ or the corresponding [design proposal](https://git.k8s.io/design-proposals-archi
`kubeadm` executes a set of preflight checks before starting the join, with the aim to verify
preconditions and avoid common cluster startup problems.
Please note that:
Also note that:
1. `kubeadm join` preflight checks are basically a subset of `kubeadm init` preflight checks
1. Starting from 1.24, kubeadm uses crictl to communicate to all known CRI endpoints.
1. Starting from 1.9, kubeadm provides support for joining nodes running on Windows; in that case,
linux specific controls are skipped.
1. If you are joining a Windows node, Linux specific controls are skipped.
1. In any case the user can skip specific preflight checks (or eventually all preflight checks)
with the `--ignore-preflight-errors` option.
@ -582,9 +598,8 @@ In order to prevent "man in the middle" attacks, several steps are taken:
compared with the CA retrieved initially
{{< note >}}
Pub key validation can be skipped passing `--discovery-token-unsafe-skip-ca-verification` flag;
This weakens the kubeadm security model since others can potentially impersonate the Kubernetes Master.
You can skip CA validation by passing the `--discovery-token-unsafe-skip-ca-verification` flag on the command line.
This weakens the kubeadm security model since others can potentially impersonate the Kubernetes API server.
{{< /note >}}
#### File/https discovery
@ -620,3 +635,88 @@ is deleted.
- The automatic CSR approval is managed by the csrapprover controller, according to
the configuration present in the `kubeadm init` process
{{< /note >}}
## kubeadm upgrade workflow internal design
`kubeadm upgrade` has sub-commands for handling the upgrade of the Kubernets cluster created by kubeadm.
You must run `kubeadm upgrade apply` on a control plane node (you can choose which one);
this starts the upgrade process. You then run `kubeadm upgrade node` on all remaining
nodes (both worker nodes and control plane nodes).
Both `kubeadm upgrade apply` and `kubeadm upgrade node` have a `phase` subcommand which provides access
to the internal phases of the upgrade process.
See [`kubeadm upgrade phase`](/docs/reference/setup-tools/kubeadm/kubeadm-upgrade-phase/) for more details.
Additional utility upgrade commands are `kubeadm upgrade plan` and `kubeadm upgrade diff`.
All upgrade sub-commands support passing a configuration file.
### kubeadm upgrade plan
You can optionally run `kubeadm upgrade plan` before you run `kubeadm upgrade apply`.
The `plan` subcommand checks which versions are available to upgrade
to and validates whether your current cluster is upgradeable.
### kubeadm upgrade diff
This shows what differences would be applied to existing static pod manifests for control plane nodes.
A more verbose way to do the same thing is running `kubeadm upgrade apply --dry-run` or
`kubeadm upgrade node --dry-run`.
### kubeadm upgrade apply
`kubeadm upgrade apply` prepares the cluster for the upgrade of all nodes, and also
upgrades the control plane node where it's run. The steps it performs are:
- Runs preflight checks similarly to `kubeadm init` and `kubeadm join`, ensuring container images are downloaded
and the cluster is in a good state to be upgraded.
- Upgrades the control plane manifest files on disk in `/etc/kubernetes/manifests` and waits
for the kubelet to restart the components if the files have changed.
- Uploads the updated kubeadm and kubelet configurations to the cluster in the `kubeadm-config`
and the `kubelet-config` ConfigMaps (both in the `kube-system` namespace).
- Writes updated kubelet configuration for this node in `/var/lib/kubelet/config.yaml`.
- Configures bootstrap token and the `cluster-info` ConfigMap for RBAC rules. This is the same as
in the `kubeadm init` stage and ensures that the cluster continues to support nodes joining with bootstrap tokens.
- Upgrades the kube-proxy and CoreDNS addons conditionally if all existing kube-apiservers in the cluster
have already been upgraded to the target version.
- Performs any post-upgrade tasks, such as, cleaning up deprecated features which are release specific.
### kubeadm upgrade node
`kubeadm upgrade node` upgrades a single control plane or worker node after the cluster upgrade has
started (by running `kubeadm upgrade apply`). The command detects if the node is a control plane node by checking
if the file `/etc/kubernetes/manifests/kube-apiserver.yaml` exists. On finding that file, the kubeadm tool
infers that there is a running kube-apiserver Pod on this node.
- Runs preflight checks similarly to `kubeadm upgrade apply`.
- For control plane nodes, upgrades the control plane manifest files on disk in `/etc/kubernetes/manifests`
and waits for the kubelet to restart the components if the files have changed.
- Writes the updated kubelet configuration for this node in `/var/lib/kubelet/config.yaml`.
- (For control plane nodes) upgrades the kube-proxy and CoreDNS
{{< glossary_tooltip text="addons" term_id="addons" >}} conditionally, provided that all existing
API servers in the cluster have already been upgraded to the target version.
- Performs any post-upgrade tasks, such as cleaning up deprecated features which are release specific.
## kubeadm reset workflow internal design
You can use the `kubeadm reset` subcommand on a node where kubeadm commands previously executed.
This subcommand performs a **best-effort** cleanup of the node.
If certain actions fail you must intervene and perform manual cleanup.
The command supports phases.
See [`kubeadm reset phase`](/docs/reference/setup-tools/kubeadm/kubeadm-reset-phase/) for more details.
The command supports a configuration file.
Additionally:
- IPVS, iptables and nftables rules are **not** cleaned up.
- CNI (network plugin) configuration is **not** cleaned up.
- `.kube/` in the user's home directory is **not** cleaned up.
The command has the following stages:
- Runs preflight checks on the node to determine if its healthy.
- For control plane nodes, removes any local etcd member data.
- Stops the kubelet.
- Stops running containers.
- Unmounts any mounted directories in `/var/lib/kubelet`.
- Deletes any files and directories managed by kubeadm in `/var/lib/kubelet` and `/etc/kubernetes`.

View File

@ -157,7 +157,7 @@ Feature | Default | Alpha | Beta | GA
:-------|:--------|:------|:-----|:----
`ControlPlaneKubeletLocalMode` | `false` | 1.31 | - | -
`EtcdLearnerMode` | `true` | 1.27 | 1.29 | 1.32
`PublicKeysECDSA` | `false` | 1.19 | - | -
`NodeLocalCRISocket` | `false` | 1.32 | - | -
`WaitForAllControlPlaneComponents` | `false` | 1.30 | - | -
{{< /table >}}
@ -176,20 +176,26 @@ policy during rolling upgrades.
: With this feature gate enabled, when joining a new control plane node, a new etcd member will be created
as a learner and promoted to a voting member only after the etcd data are fully aligned.
`PublicKeysECDSA`
: Can be used to create a cluster that uses ECDSA certificates instead of the default RSA algorithm.
Renewal of existing ECDSA certificates is also supported using `kubeadm certs renew`, but you cannot
switch between the RSA and ECDSA algorithms on the fly or during upgrades. Kubernetes
{{< skew currentVersion >}} has a bug where keys in generated kubeconfig files are set use RSA
despite the feature gate being enabled. Kubernetes versions before v1.31 had a bug where keys in generated kubeconfig files
were set use RSA, even when you had enabled the `PublicKeysECDSA` feature gate.
`NodeLocalCRISocket`
: With this feature gate enabled, kubeadm will read/write the CRI socket for each node from/to the file
`/var/lib/kubelet/instance-config.yaml` instead of reading/writing it from/to the annotation
`kubeadm.alpha.kubernetes.io/cri-socket` on the Node object. The new file is applied as an instance
configuration patch, before any other user managed patches are applied when the `--patches` flag
is used. It contains a single field `containerRuntimeEndpoint` from the
[KubeletConfiguration file format](/docs/reference/config-api/kubelet-config.v1beta1/). If the feature gate
is enabled during upgrade, but the file `/var/lib/kubelet/instance-config.yaml` does not exist yet,
kubeadm will attempt to read the CRI socket value from the file `/var/lib/kubelet/kubeadm-flags.env`.
`WaitForAllControlPlaneComponents`
: With this feature gate enabled kubeadm will wait for all control plane components (kube-apiserver,
kube-controller-manager, kube-scheduler) on a control plane node to report status 200 on their `/healthz`
endpoints. These checks are performed on `https://127.0.0.1:PORT/healthz`, where `PORT` is taken from
`--secure-port` of a component. If you specify custom `--secure-port` values in the kubeadm configuration
they will be respected. Without the feature gate enabled, kubeadm will only wait for the kube-apiserver
: With this feature gate enabled, kubeadm will wait for all control plane components (kube-apiserver,
kube-controller-manager, kube-scheduler) on a control plane node to report status 200 on their `/livez`
or `/healthz` endpoints. These checks are performed on `https://ADDRESS:PORT/ENDPOINT`.
- `PORT` is taken from `--secure-port` of a component.
- `ADDRESS` is `--advertise-address` for kube-apiserver and `--bind-address` for the kube-controller-manager
and kube-scheduler.
- `ENDPOINT` is only `/healthz` for kube-controller-manager until it supports `/livez` as well.
If you specify custom `ADDRESS` or `PORT` in the kubeadm configuration they will be respected.
Without the feature gate enabled, kubeadm will only wait for the kube-apiserver
on a control plane node to become ready. The wait process starts right after the kubelet on the host
is started by kubeadm. You are advised to enable this feature gate in case you wish to observe a ready
state from all control plane components during the `kubeadm init` or `kubeadm join` command execution.
@ -199,11 +205,20 @@ List of deprecated feature gates:
{{< table caption="kubeadm deprecated feature gates" >}}
Feature | Default | Alpha | Beta | GA | Deprecated
:-------|:--------|:------|:-----|:---|:----------
`PublicKeysECDSA` | `false` | 1.19 | - | - | 1.31
`RootlessControlPlane` | `false` | 1.22 | - | - | 1.31
{{< /table >}}
Feature gate descriptions:
`PublicKeysECDSA`
: Can be used to create a cluster that uses ECDSA certificates instead of the default RSA algorithm.
Renewal of existing ECDSA certificates is also supported using `kubeadm certs renew`, but you cannot
switch between the RSA and ECDSA algorithms on the fly or during upgrades.Kubernetes versions before v1.31
had a bug where keys in generated kubeconfig files were set use RSA, even when you had enabled the
`PublicKeysECDSA` feature gate. This feature gate is deprecated in favor of the `encryptionAlgorithm`
functionality available in kubeadm v1beta4.
`RootlessControlPlane`
: Setting this flag configures the kubeadm deployed control plane component static Pod containers
for `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and `etcd` to run as non-root users.

View File

@ -1,23 +1,37 @@
---
title: kubeadm upgrade phase
weight: 90
title: kubeadm upgrade phases
weight: 40
content_type: concept
---
In v1.15.0, kubeadm introduced preliminary support for `kubeadm upgrade node` phases.
Phases for other `kubeadm upgrade` sub-commands such as `apply`, could be added in the
following releases.
## kubeadm upgrade apply phase {#cmd-apply-phase}
Using the phases of `kubeadm upgrade apply`, you can choose to execute the separate steps of the initial upgrade
of a control plane node.
{{< tabs name="tab-phase" >}}
{{< tab name="phase" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase.md" />}}
{{< tab name="preflight" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_preflight.md" />}}
{{< tab name="control-plane" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_control-plane.md" />}}
{{< tab name="upload-config" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_upload-config.md" />}}
{{< tab name="kubelet-config" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_kubelet-config.md" />}}
{{< tab name="bootstrap-token" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_bootstrap-token.md" />}}
{{< tab name="addon" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_addon.md" />}}
{{< tab name="post-upgrade" include="generated/kubeadm_upgrade/kubeadm_upgrade_apply_phase_post-upgrade.md" />}}
{{< /tabs >}}
## kubeadm upgrade node phase {#cmd-node-phase}
Using this phase you can choose to execute the separate steps of the upgrade of
secondary control-plane or worker nodes. Please note that `kubeadm upgrade apply` still has to
be called on a primary control-plane node.
Using the phases of `kubeadm upgrade node` you can choose to execute the separate steps of the upgrade of
secondary control-plane or worker nodes.
{{< tabs name="tab-phase" >}}
{{< tab name="phase" include="generated/kubeadm_upgrade/kubeadm_upgrade_node_phase.md" />}}
{{< tab name="preflight" include="generated/kubeadm_upgrade/kubeadm_upgrade_node_phase_preflight.md" />}}
{{< tab name="control-plane" include="generated/kubeadm_upgrade/kubeadm_upgrade_node_phase_control-plane.md" />}}
{{< tab name="kubelet-config" include="generated/kubeadm_upgrade/kubeadm_upgrade_node_phase_kubelet-config.md" />}}
{{< tab name="addon" include="generated/kubeadm_upgrade/kubeadm_upgrade_node_phase_addon.md" />}}
{{< tab name="post-upgrade" include="generated/kubeadm_upgrade/kubeadm_upgrade_node_phase_post-upgrade.md" />}}
{{< /tabs >}}
## {{% heading "whatsnext" %}}

View File

@ -24,6 +24,8 @@ Following this recommendation helps you to to stay secure.
You should be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/best-practices/certificates/).
You should be familiar with how to pass a [configuration](/docs/reference/config-api/kubeadm-config.v1beta4/) file to the kubeadm commands.
This guide covers the usage of the `openssl` command (used for manual certificate signing,
if you choose that approach), but you can use your preferred tools.
@ -45,6 +47,37 @@ kubeadm does not overwrite them. This means you can, for example, copy an existi
CA into `/etc/kubernetes/pki/ca.crt` and `/etc/kubernetes/pki/ca.key`,
and kubeadm will use this CA for signing the rest of the certificates.
## Choosing an encryption algorithm {#choosing-encryption-algorithm}
kubeadm allows you to choose an encryption algorithm that is used for creating
public and private keys. That can be done by using the `encryptionAlgorithm` field of the
kubeadm configuration:
```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
encryptionAlgorithm: <ALGORITHM>
```
`<ALGORITHM>` can be one of `RSA-2048` (default), `RSA-3072`, `RSA-4096` or `ECDSA-P256`.
## Choosing certificate validity period {#choosing-cert-validity-period}
kubeadm allows you to choose the validity period of CA and leaf certificates.
That can be done by using the `certificateValidityPeriod` and `caCertificateValidityPeriod`
fields of the kubeadm configuration:
```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
certificateValidityPeriod: 8760h # Default: 365 days × 24 hours = 1 year
caCertificateValidityPeriod: 87600h # Default: 365 days × 24 hours * 10 = 10 years
```
The values of the fields follow the accepted format for
[Go's `time.Duration` values](https://pkg.go.dev/time#ParseDuration), with the longest supported
unit being `h` (hours).
## External CA mode {#external-ca-mode}
It is also possible to provide only the `ca.crt` file and not the