Update Troubleshooting kubeadm

pull/22631/head
inductor 2020-07-22 07:36:42 +09:00
parent 189444078a
commit 86010e1cf3
1 changed files with 49 additions and 8 deletions

View File

@ -1,7 +1,7 @@
--- ---
title: kubeadmのトラブルシューティング title: kubeadmのトラブルシューティング
content_type: concept content_type: concept
weight: 90 weight: 20
--- ---
<!-- overview --> <!-- overview -->
@ -152,7 +152,7 @@ Unable to connect to the server: x509: certificate signed by unknown authority (
- Verify that the `$HOME/.kube/config` file contains a valid certificate, and - Verify that the `$HOME/.kube/config` file contains a valid certificate, and
regenerate a certificate if necessary. The certificates in a kubeconfig file regenerate a certificate if necessary. The certificates in a kubeconfig file
are base64 encoded. The `base64 -d` command can be used to decode the certificate are base64 encoded. The `base64 --decode` command can be used to decode the certificate
and `openssl x509 -text -noout` can be used for viewing the certificate information. and `openssl x509 -text -noout` can be used for viewing the certificate information.
- Unset the `KUBECONFIG` environment variable using: - Unset the `KUBECONFIG` environment variable using:
@ -170,6 +170,7 @@ Unable to connect to the server: x509: certificate signed by unknown authority (
```sh ```sh
mv $HOME/.kube $HOME/.kube.bak mv $HOME/.kube $HOME/.kube.bak
mkdir $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
``` ```
@ -197,15 +198,15 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6
``` ```
- This may be due to Kubernetes using an IP that can not communicate with other IPs on the seemingly same subnet, possibly by policy of the machine provider. - This may be due to Kubernetes using an IP that can not communicate with other IPs on the seemingly same subnet, possibly by policy of the machine provider.
- Digital Ocean assigns a public IP to `eth0` as well as a private one to be used internally as anchor for their floating IP feature, yet `kubelet` will pick the latter as the node's `InternalIP` instead of the public one. - DigitalOcean assigns a public IP to `eth0` as well as a private one to be used internally as anchor for their floating IP feature, yet `kubelet` will pick the latter as the node's `InternalIP` instead of the public one.
Use `ip addr show` to check for this scenario instead of `ifconfig` because `ifconfig` will not display the offending alias IP address. Alternatively an API endpoint specific to Digital Ocean allows to query for the anchor IP from the droplet: Use `ip addr show` to check for this scenario instead of `ifconfig` because `ifconfig` will not display the offending alias IP address. Alternatively an API endpoint specific to DigitalOcean allows to query for the anchor IP from the droplet:
```sh ```sh
curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address
``` ```
The workaround is to tell `kubelet` which IP to use using `--node-ip`. When using Digital Ocean, it can be the public one (assigned to `eth0`) or the private one (assigned to `eth1`) should you want to use the optional private network. The [`KubeletExtraArgs` section of the kubeadm `NodeRegistrationOptions` structure](https://github.com/kubernetes/kubernetes/blob/release-1.13/cmd/kubeadm/app/apis/kubeadm/v1beta1/types.go) can be used for this. The workaround is to tell `kubelet` which IP to use using `--node-ip`. When using DigitalOcean, it can be the public one (assigned to `eth0`) or the private one (assigned to `eth1`) should you want to use the optional private network. The [`KubeletExtraArgs` section of the kubeadm `NodeRegistrationOptions` structure](https://github.com/kubernetes/kubernetes/blob/release-1.13/cmd/kubeadm/app/apis/kubeadm/v1beta1/types.go) can be used for this.
Then restart `kubelet`: Then restart `kubelet`:
@ -306,16 +307,56 @@ The tracking issue for this problem is [here](https://github.com/kubernetes/kube
*Note: This [issue](https://github.com/kubernetes/kubeadm/issues/1358) only applies to tools that marshal kubeadm types (e.g. to a YAML configuration file). It will be fixed in kubeadm API v1beta2.* *Note: This [issue](https://github.com/kubernetes/kubeadm/issues/1358) only applies to tools that marshal kubeadm types (e.g. to a YAML configuration file). It will be fixed in kubeadm API v1beta2.*
By default, kubeadm applies the `role.kubernetes.io/master:NoSchedule` taint to control-plane nodes. By default, kubeadm applies the `node-role.kubernetes.io/master:NoSchedule` taint to control-plane nodes.
If you prefer kubeadm to not taint the control-plane node, and set `InitConfiguration.NodeRegistration.Taints` to an empty slice, If you prefer kubeadm to not taint the control-plane node, and set `InitConfiguration.NodeRegistration.Taints` to an empty slice,
the field will be omitted when marshalling. When the field is omitted, kubeadm applies the default taint. the field will be omitted when marshalling. When the field is omitted, kubeadm applies the default taint.
There are at least two workarounds: There are at least two workarounds:
1. Use the `role.kubernetes.io/master:PreferNoSchedule` taint instead of an empty slice. [Pods will get scheduled on masters](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/), unless other nodes have capacity. 1. Use the `node-role.kubernetes.io/master:PreferNoSchedule` taint instead of an empty slice. [Pods will get scheduled on masters](/docs/concepts/configuration/taint-and-toleration/), unless other nodes have capacity.
2. Remove the taint after kubeadm init exits: 2. Remove the taint after kubeadm init exits:
```bash ```bash
kubectl taint nodes NODE_NAME role.kubernetes.io/master:NoSchedule- kubectl taint nodes NODE_NAME node-role.kubernetes.io/master:NoSchedule-
```
## `/usr` is mounted read-only on nodes {#usr-mounted-read-only}
On Linux distributions such as Fedora CoreOS, the directory `/usr` is mounted as a read-only filesystem.
For [flex-volume support](https://github.com/kubernetes/community/blob/ab55d85/contributors/devel/sig-storage/flexvolume.md),
Kubernetes components like the kubelet and kube-controller-manager use the default path of
`/usr/libexec/kubernetes/kubelet-plugins/volume/exec/`, yet the flex-volume directory _must be writeable_
for the feature to work.
To workaround this issue you can configure the flex-volume directory using the kubeadm
[configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2).
On the primary control-plane Node (created using `kubeadm init`) pass the following
file using `--config`:
```yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controllerManager:
extraArgs:
flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
``` ```
On joining Nodes:
```yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
```
Alternatively, you can modify `/etc/fstab` to make the `/usr` mount writeable, but please
be advised that this is modifying a design principle of the Linux distribution.