Add CoreDNS details to DNS Debug docs (#10201)
* add coredns details * address nits, add query logging sectionpull/10064/head
Chris O'Haver
k8s-ci-robot
parent
c98cd68630
commit
462817a674
|
|
@ -13,7 +13,7 @@ This page provides hints on diagnosing DNS problems.
|
|||
{{% capture prerequisites %}}
|
||||
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
||||
* Kubernetes version 1.6 and above.
|
||||
* The cluster must be configured to use the `kube-dns` addon.
|
||||
* The cluster must be configured to use the `coredns` (or `kube-dns`) addons.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
|
@ -68,7 +68,7 @@ nameserver 10.0.0.10
|
|||
options ndots:5
|
||||
```
|
||||
|
||||
Errors such as the following indicate a problem with the kube-dns add-on or
|
||||
Errors such as the following indicate a problem with the coredns/kube-dns add-on or
|
||||
associated Services:
|
||||
|
||||
```
|
||||
|
|
@ -93,6 +93,17 @@ nslookup: can't resolve 'kubernetes.default'
|
|||
|
||||
Use the `kubectl get pods` command to verify that the DNS pod is running.
|
||||
|
||||
For CoreDNS:
|
||||
```shell
|
||||
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
...
|
||||
coredns-7b96bf9f76-5hsxb 1/1 Running 0 1h
|
||||
coredns-7b96bf9f76-mvmmt 1/1 Running 0 1h
|
||||
...
|
||||
```
|
||||
|
||||
Or for kube-dns:
|
||||
```shell
|
||||
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
|
|
@ -107,8 +118,26 @@ have to deploy it manually.
|
|||
|
||||
### Check for Errors in the DNS pod
|
||||
|
||||
Use `kubectl logs` command to see logs for the DNS daemons.
|
||||
Use `kubectl logs` command to see logs for the DNS containers.
|
||||
|
||||
For CoreDNS:
|
||||
```shell
|
||||
for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
|
||||
```
|
||||
|
||||
Here is an example of a healthy CoreDNS log:
|
||||
|
||||
```
|
||||
.:53
|
||||
2018/08/15 14:37:17 [INFO] CoreDNS-1.2.2
|
||||
2018/08/15 14:37:17 [INFO] linux/amd64, go1.10.3, 2e322f6
|
||||
CoreDNS-1.2.2
|
||||
linux/amd64, go1.10.3, 2e322f6
|
||||
2018/08/15 14:37:17 [INFO] plugin/reload: Running configuration MD5 = 24e6c59e83ce706f07bcc82c31b1ea1c
|
||||
```
|
||||
|
||||
|
||||
For kube-dns, there are 3 sets of logs:
|
||||
```shell
|
||||
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c kubedns
|
||||
|
||||
|
|
@ -117,8 +146,8 @@ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system
|
|||
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c sidecar
|
||||
```
|
||||
|
||||
See if there is any suspicious log. Letter '`W`', '`E`', '`F`' at the beginning
|
||||
represent Warning, Error and Failure. Please search for entries that have these
|
||||
See if there are any suspicious error messages in the logs. In kube-dns, a '`W`', '`E`' or '`F`' at the beginning
|
||||
of a line represents a Warning, Error or Failure. Please search for entries that have these
|
||||
as the logging level and use
|
||||
[kubernetes issues](https://github.com/kubernetes/kubernetes/issues)
|
||||
to report unexpected errors.
|
||||
|
|
@ -135,6 +164,8 @@ kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 1h
|
|||
...
|
||||
```
|
||||
|
||||
|
||||
Note that the service name will be "kube-dns" for both CoreDNS and kube-dns deployments.
|
||||
If you have created the service or in the case it should be created by default
|
||||
but it does not appear, see
|
||||
[debugging services](/docs/tasks/debug-application-cluster/debug-service/) for
|
||||
|
|
@ -158,20 +189,83 @@ For additional Kubernetes DNS examples, see the
|
|||
[cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns)
|
||||
in the Kubernetes GitHub repository.
|
||||
|
||||
|
||||
### Are DNS queries being received/processed?
|
||||
|
||||
You can verify if queries are being received by CoreDNS by adding the `log` plugin to the CoreDNS configuration (aka Corefile).
|
||||
The CoreDNS Corefile is held in a ConfigMap named `coredns`. To edit it, use the command ...
|
||||
|
||||
```
|
||||
kubectl -n kube-system edit configmap coredns
|
||||
```
|
||||
|
||||
Then add `log` in the Corefile section per the example below.
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: coredns
|
||||
namespace: kube-system
|
||||
data:
|
||||
Corefile: |
|
||||
.:53 {
|
||||
log
|
||||
errors
|
||||
health
|
||||
kubernetes cluster.local in-addr.arpa ip6.arpa {
|
||||
pods insecure
|
||||
upstream
|
||||
fallthrough in-addr.arpa ip6.arpa
|
||||
}
|
||||
prometheus :9153
|
||||
proxy . /etc/resolv.conf
|
||||
cache 30
|
||||
loop
|
||||
reload
|
||||
loadbalance
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
After saving the changes, it may take up to minute or two for Kubernetes to propagate these changes to the CoreDNS pods.
|
||||
|
||||
Next, make some queries and view the logs per the sections above in this document. If CoreDNS pods are receiving the queries, you should see them in the logs.
|
||||
|
||||
Here is an example of a query in the log.
|
||||
|
||||
```
|
||||
.:53
|
||||
2018/08/15 14:37:15 [INFO] CoreDNS-1.2.0
|
||||
2018/08/15 14:37:15 [INFO] linux/amd64, go1.10.3, 2e322f6
|
||||
CoreDNS-1.2.0
|
||||
linux/amd64, go1.10.3, 2e322f6
|
||||
2018/09/07 15:29:04 [INFO] plugin/reload: Running configuration MD5 = 162475cdf272d8aa601e6fe67a6ad42f
|
||||
2018/09/07 15:29:04 [INFO] Reloading complete
|
||||
172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s
|
||||
|
||||
```
|
||||
|
||||
## Known issues
|
||||
|
||||
Kubernetes installs do not configure the nodes' resolv.conf files to use the
|
||||
cluster DNS by default, because that process is inherently distro-specific.
|
||||
Some Linux distributions (e.g. Ubuntu), use a local DNS resolver by default (systemd-resolved).
|
||||
Systemd-resolved moves and replaces `/etc/resolv.conf` with a stub file that can cause a fatal forwarding
|
||||
loop when resolving names in upstream servers. This can be fixed manually by using kubelet's `--resolv-conf` flag
|
||||
to point to the correct `resolv.conf` (With `systemd-resolved`, this is `/run/systemd/resolve/resolv.conf`).
|
||||
kubeadm 1.11 automatically detects `systemd-resolved`, and adjusts the kubelet flags accordingly.
|
||||
|
||||
Kubernetes installs do not configure the nodes' `resolv.conf` files to use the
|
||||
cluster DNS by default, because that process is inherently distribution-specific.
|
||||
This should probably be implemented eventually.
|
||||
|
||||
Linux's libc is impossibly stuck ([see this bug from
|
||||
2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just
|
||||
3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to
|
||||
consume 1 `nameserver` record and 3 `search` records. This means that if a
|
||||
3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to
|
||||
consume 1 `nameserver` record and 3 `search` records. This means that if a
|
||||
local installation already uses 3 `nameserver`s or uses more than 3 `search`es,
|
||||
some of those settings will be lost. As a partial workaround, the node can run
|
||||
some of those settings will be lost. As a partial workaround, the node can run
|
||||
`dnsmasq` which will provide more `nameserver` entries, but not more `search`
|
||||
entries. You can also use kubelet's `--resolv-conf` flag.
|
||||
entries. You can also use kubelet's `--resolv-conf` flag.
|
||||
|
||||
If you are using Alpine version 3.3 or earlier as your base image, DNS may not
|
||||
work properly owing to a known issue with Alpine.
|
||||
|
|
|
|||
Loading…
Reference in New Issue