144 lines
6.2 KiB
Markdown
144 lines
6.2 KiB
Markdown
|
---
|
|||
|
|
title: Troubleshooting kubeadm
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
{% capture overview %}
|
|||
|
|
|
|||
|
|
As with any program, you might run into an error using or operating it. Below we have listed
|
|||
|
|
common failure scenarios and have provided steps that will help you to understand and hopefully
|
|||
|
|
fix the problem.
|
|||
|
|
|
|||
|
|
If your problem is not listed below, please follow the following steps:
|
|||
|
|
|
|||
|
|
- If you think your problem is a bug with kubeadm:
|
|||
|
|
- Go to [github.com/kubernetes/kubeadm](https://github.com/kubernetes/kubeadm/issues) and search for existing issues.
|
|||
|
|
- If no issue exists, please [open one](https://github.com/kubernetes/kubeadm/issues/new) and follow the issue template.
|
|||
|
|
|
|||
|
|
- If you are unsure about how kubeadm or kubernetes works, and would like to receive
|
|||
|
|
support about your question, please ask on Slack in #kubeadm, or open a question on StackOverflow. Please include
|
|||
|
|
relevant tags like `#kubernetes` and `#kubeadm` so folks can help you.
|
|||
|
|
|
|||
|
|
If your cluster is in an error state, you may have trouble in the configuration if you see Pod statuses like `RunContainerError`,
|
|||
|
|
`CrashLoopBackOff` or `Error`. If this is the case, please read below.
|
|||
|
|
|
|||
|
|
{% endcapture %}
|
|||
|
|
|
|||
|
|
#### `ebtables` or `ethtool` not found during installation
|
|||
|
|
|
|||
|
|
If you see the following warnings while running `kubeadm init`
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
[preflight] WARNING: ebtables not found in system path
|
|||
|
|
[preflight] WARNING: ethtool not found in system path
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Then you may be missing ebtables and ethtool on your Linux machine. You can install them with the following commands:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
# For ubuntu/debian users, try
|
|||
|
|
apt install ebtables ethtool
|
|||
|
|
|
|||
|
|
# For CentOS/Fedora users, try
|
|||
|
|
yum install ebtables ethtool
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
#### Pods in `RunContainerError`, `CrashLoopBackOff` or `Error` state
|
|||
|
|
|
|||
|
|
Right after `kubeadm init` there should not be any such Pods. If there are Pods in
|
|||
|
|
such a state _right after_ `kubeadm init`, please open an issue in the kubeadm repo.
|
|||
|
|
`kube-dns` should be in the `Pending` state until you have deployed the network solution.
|
|||
|
|
However, if you see Pods in the `RunContainerError`, `CrashLoopBackOff` or `Error` state
|
|||
|
|
after deploying the network solution and nothing happens to `kube-dns`, it's very
|
|||
|
|
likely that the Pod Network solution that you installed is somehow broken. You
|
|||
|
|
might have to grant it more RBAC privileges or use a newer version. Please file
|
|||
|
|
an issue in the Pod Network providers' issue tracker and get the issue triaged there.
|
|||
|
|
|
|||
|
|
#### `kube-dns` is stuck in the `Pending` state
|
|||
|
|
|
|||
|
|
This is **expected** and part of the design. kubeadm is network provider-agnostic, so the admin
|
|||
|
|
should [install the pod network solution](/docs/concepts/cluster-administration/addons/)
|
|||
|
|
of choice. You have to install a Pod Network
|
|||
|
|
before `kube-dns` may deployed fully. Hence the `Pending` state before the network is set up.
|
|||
|
|
|
|||
|
|
#### `HostPort` services do not work
|
|||
|
|
|
|||
|
|
The `HostPort` and `HostIP` functionality is available depending on your Pod Network
|
|||
|
|
provider. Please contact the author of the Pod Network solution to find out whether
|
|||
|
|
`HostPort` and `HostIP` functionality are available.
|
|||
|
|
|
|||
|
|
Verified HostPort CNI providers:
|
|||
|
|
- Calico
|
|||
|
|
- Canal
|
|||
|
|
- Flannel
|
|||
|
|
|
|||
|
|
For more information, read the [CNI portmap documentation](https://github.com/containernetworking/plugins/blob/master/plugins/meta/portmap/README.md).
|
|||
|
|
|
|||
|
|
If your network provider does not support the portmap CNI plugin, you may need to use the [NodePort feature of
|
|||
|
|
services](/docs/concepts/services-networking/service/#type-nodeport) or use `HostNetwork=true`.
|
|||
|
|
|
|||
|
|
#### Pods are not accessible via their Service IP
|
|||
|
|
|
|||
|
|
Many network add-ons do not yet enable [hairpin mode](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#a-pod-cannot-reach-itself-via-service-ip)
|
|||
|
|
which allows pods to access themselves via their Service IP if they don't know about their podIP. This is an issue
|
|||
|
|
related to [CNI](https://github.com/containernetworking/cni/issues/476). Please contact the providers of the network
|
|||
|
|
add-on providers to get timely information about whether they support hairpin mode.
|
|||
|
|
|
|||
|
|
If you are using VirtualBox (directly or via Vagrant), you will need to
|
|||
|
|
ensure that `hostname -i` returns a routable IP address (i.e. one on the
|
|||
|
|
second network interface, not the first one). By default, it doesn't do this
|
|||
|
|
and kubelet ends-up using first non-loopback network interface, which is
|
|||
|
|
usually NATed. Workaround: Modify `/etc/hosts`, take a look at this
|
|||
|
|
`Vagrantfile`[ubuntu-vagrantfile](https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11) for how this can be achieved.
|
|||
|
|
|
|||
|
|
#### TLS certificate errors
|
|||
|
|
|
|||
|
|
The following error indicates a possible certificate mismatch.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
# kubectl get po
|
|||
|
|
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Verify that the `$HOME/.kube/config` file contains a valid certificate, and regenerate a certificate if necessary.
|
|||
|
|
Another workaround is to overwrite the default `kubeconfig` for the "admin" user:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
mv $HOME/.kube $HOME/.kube.bak
|
|||
|
|
mkdir -p $HOME/.kube
|
|||
|
|
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
|
|||
|
|
sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
#### Errors on CentOS when setting up masters
|
|||
|
|
|
|||
|
|
If you are using CentOS and encounter difficulty while setting up the master node，
|
|||
|
|
verify that your Docker cgroup driver matches the kubelet config:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
docker info | grep -i cgroup
|
|||
|
|
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
If the Docker cgroup driver and the kubelet config don't match, change the kubelet config to match the Docker cgroup driver. The
|
|||
|
|
flag you need to change is `--cgroup-driver`. If it's already set, you can update like so:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Otherwise, you will need to open the systemd file and add the flag to an existing environment line.
|
|||
|
|
|
|||
|
|
Then restart kubelet:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
systemctl daemon-reload
|
|||
|
|
systemctl restart kubelet
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The `kubectl describe pod` or `kubectl logs` commands can help you diagnose errors. For example:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
kubectl -n ${NAMESPACE} describe pod ${POD_NAME}
|
|||
|
|
|
|||
|
|
kubectl -n ${NAMESPACE} logs ${POD_NAME} -c ${CONTAINER_NAME}
|
|||
|
|
```
|