Unify debug pod docs and update for `kubectl alpha debug` (#19093)
Co-Authored-By: Tim Bannister <tim@scalefactory.com> Co-authored-by: Tim Bannister <tim@scalefactory.com>pull/19349/head
Lee Verberne
GitHub
parent
e752b37bf0
commit
46bda9606a
|
|
@ -12,16 +12,15 @@ weight: 80
|
|||
{{< feature-state state="alpha" for_k8s_version="v1.16" >}}
|
||||
|
||||
This page provides an overview of ephemeral containers: a special type of container
|
||||
that runs temporarily in an existing {{< glossary_tooltip term_id="pod" >}} to accomplish user-initiated actions such
|
||||
as troubleshooting. You use ephemeral containers to inspect services rather than
|
||||
to build applications.
|
||||
that runs temporarily in an existing {{< glossary_tooltip term_id="pod" >}} to
|
||||
accomplish user-initiated actions such as troubleshooting. You use ephemeral
|
||||
containers to inspect services rather than to build applications.
|
||||
|
||||
{{< warning >}}
|
||||
Ephemeral containers are in early alpha state and are not suitable for production
|
||||
clusters. You should expect the feature not to work in some situations, such as
|
||||
when targeting the namespaces of a container. In accordance with the [Kubernetes
|
||||
Deprecation Policy](/docs/reference/using-api/deprecation-policy/), this alpha
|
||||
feature could change significantly in the future or be removed entirely.
|
||||
clusters. In accordance with the [Kubernetes Deprecation Policy](
|
||||
/docs/reference/using-api/deprecation-policy/), this alpha feature could change
|
||||
significantly in the future or be removed entirely.
|
||||
{{< /warning >}}
|
||||
|
||||
{{% /capture %}}
|
||||
|
|
@ -78,7 +77,11 @@ When using ephemeral containers, it's helpful to enable [process namespace
|
|||
sharing](/docs/tasks/configure-pod-container/share-process-namespace/) so
|
||||
you can view processes in other containers.
|
||||
|
||||
### Examples
|
||||
See [Debugging with Ephemeral Debug Container](
|
||||
/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container)
|
||||
for examples of troubleshooting using ephemeral containers.
|
||||
|
||||
## Ephemeral containers API
|
||||
|
||||
{{< note >}}
|
||||
The examples in this section require the `EphemeralContainers` [feature
|
||||
|
|
@ -87,8 +90,9 @@ enabled, and Kubernetes client and server version v1.16 or later.
|
|||
{{< /note >}}
|
||||
|
||||
The examples in this section demonstrate how ephemeral containers appear in
|
||||
the API. You would normally use a `kubectl` plugin for troubleshooting that
|
||||
automates these steps.
|
||||
the API. You would normally use `kubectl alpha debug` or another `kubectl`
|
||||
[plugin](/docs/tasks/extend-kubectl/kubectl-plugins/) to automate these steps
|
||||
rather than invoking the API directly.
|
||||
|
||||
Ephemeral containers are created using the `ephemeralcontainers` subresource
|
||||
of Pod, which can be demonstrated using `kubectl --raw`. First describe
|
||||
|
|
@ -180,35 +184,12 @@ Ephemeral Containers:
|
|||
...
|
||||
```
|
||||
|
||||
You can attach to the new ephemeral container using `kubectl attach`:
|
||||
You can interact with the new ephemeral container in the same way as other
|
||||
containers using `kubectl attach`, `kubectl exec`, and `kubectl logs`, for
|
||||
example:
|
||||
|
||||
```shell
|
||||
kubectl attach -it example-pod -c debugger
|
||||
```
|
||||
|
||||
If process namespace sharing is enabled, you can see processes from all the containers in that Pod.
|
||||
For example, after attaching, you run `ps` in the debugger container:
|
||||
|
||||
```shell
|
||||
# Run this in a shell inside the "debugger" ephemeral container
|
||||
ps auxww
|
||||
```
|
||||
The output is similar to:
|
||||
```
|
||||
PID USER TIME COMMAND
|
||||
1 root 0:00 /pause
|
||||
6 root 0:00 nginx: master process nginx -g daemon off;
|
||||
11 101 0:00 nginx: worker process
|
||||
12 101 0:00 nginx: worker process
|
||||
13 101 0:00 nginx: worker process
|
||||
14 101 0:00 nginx: worker process
|
||||
15 101 0:00 nginx: worker process
|
||||
16 101 0:00 nginx: worker process
|
||||
17 101 0:00 nginx: worker process
|
||||
18 101 0:00 nginx: worker process
|
||||
19 root 0:00 /pause
|
||||
24 root 0:00 sh
|
||||
29 root 0:00 ps auxww
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
|
|
|||
|
|
@ -64,38 +64,8 @@ Again, the information from `kubectl describe ...` should be informative. The m
|
|||
|
||||
#### My pod is crashing or otherwise unhealthy
|
||||
|
||||
First, take a look at the logs of
|
||||
the current container:
|
||||
|
||||
```shell
|
||||
kubectl logs ${POD_NAME} ${CONTAINER_NAME}
|
||||
```
|
||||
|
||||
If your container has previously crashed, you can access the previous container's crash log with:
|
||||
|
||||
```shell
|
||||
kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
|
||||
```
|
||||
|
||||
Alternately, you can run commands inside that container with `exec`:
|
||||
|
||||
```shell
|
||||
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
`-c ${CONTAINER_NAME}` is optional. You can omit it for Pods that only contain a single container.
|
||||
{{< /note >}}
|
||||
|
||||
As an example, to look at the logs from a running Cassandra pod, you might run
|
||||
|
||||
```shell
|
||||
kubectl exec cassandra -- cat /var/log/cassandra/system.log
|
||||
```
|
||||
|
||||
If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host,
|
||||
but this should generally not be necessary given tools in the Kubernetes API. Therefore, if you find yourself needing to ssh into a machine, please file a
|
||||
feature request on GitHub describing your use case and why these tools are insufficient.
|
||||
Once your pod has been scheduled, the methods described in [Debug Running Pods](
|
||||
/docs/tasks/debug-application-cluster/debug-running-pods/) are available for debugging.
|
||||
|
||||
#### My pod is running but not doing what I told it to do
|
||||
|
||||
|
|
|
|||
|
|
@ -93,40 +93,9 @@ worker node, but it can't run on that machine. Again, the information from
|
|||
|
||||
### My pod is crashing or otherwise unhealthy
|
||||
|
||||
First, take a look at the logs of the current container:
|
||||
Once your pod has been scheduled, the methods described in [Debug Running Pods](
|
||||
/docs/tasks/debug-application-cluster/debug-running-pods/) are available for debugging.
|
||||
|
||||
```shell
|
||||
kubectl logs ${POD_NAME} ${CONTAINER_NAME}
|
||||
```
|
||||
|
||||
If your container has previously crashed, you can access the previous
|
||||
container's crash log with:
|
||||
|
||||
```shell
|
||||
kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
|
||||
```
|
||||
|
||||
Alternately, you can run commands inside that container with `exec`:
|
||||
|
||||
```shell
|
||||
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
`-c ${CONTAINER_NAME}` is optional. You can omit it for pods that
|
||||
only contain a single container.
|
||||
{{< /note >}}
|
||||
|
||||
As an example, to look at the logs from a running Cassandra pod, you might run:
|
||||
|
||||
```shell
|
||||
kubectl exec cassandra -- cat /var/log/cassandra/system.log
|
||||
```
|
||||
|
||||
If your cluster enabled it, you can also try adding an [ephemeral container](/docs/concepts/workloads/pods/ephemeral-containers/) into the existing pod. You can use the new temporary container to run arbitrary commands, for example, to diagnose problems inside the Pod. See the page about [ephemeral container](/docs/concepts/workloads/pods/ephemeral-containers/) for more details, including feature availability.
|
||||
|
||||
If none of these approaches work, you can find the host machine that the pod is
|
||||
running on and SSH into that host.
|
||||
|
||||
## Debugging ReplicationControllers
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,190 @@
|
|||
---
|
||||
reviewers:
|
||||
- verb
|
||||
- soltysh
|
||||
title: Debug Running Pods
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
This page explains how to debug Pods running (or crashing) on a Node.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
* Your {{< glossary_tooltip text="Pod" term_id="pod" >}} should already be
|
||||
scheduled and running. If your Pod is not yet running, start with [Troubleshoot
|
||||
Applications](/docs/tasks/debug-application-cluster/debug-application/).
|
||||
* For some of the advanced debugging steps you need to know on which Node the
|
||||
Pod is running and have shell access to run commands on that Node. You don't
|
||||
need that access to run the standard debug steps that use `kubectl`.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Examining pod logs {#examine-pod-logs}
|
||||
|
||||
First, look at the logs of the affected container:
|
||||
|
||||
```shell
|
||||
kubectl logs ${POD_NAME} ${CONTAINER_NAME}
|
||||
```
|
||||
|
||||
If your container has previously crashed, you can access the previous container's crash log with:
|
||||
|
||||
```shell
|
||||
kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
|
||||
```
|
||||
|
||||
## Debugging with container exec {#container-exec}
|
||||
|
||||
If the {{< glossary_tooltip text="container image" term_id="image" >}} includes
|
||||
debugging utilities, as is the case with images built from Linux and Windows OS
|
||||
base images, you can run commands inside a specific container with
|
||||
`kubectl exec`:
|
||||
|
||||
```shell
|
||||
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
`-c ${CONTAINER_NAME}` is optional. You can omit it for Pods that only contain a single container.
|
||||
{{< /note >}}
|
||||
|
||||
As an example, to look at the logs from a running Cassandra pod, you might run
|
||||
|
||||
```shell
|
||||
kubectl exec cassandra -- cat /var/log/cassandra/system.log
|
||||
```
|
||||
|
||||
You can run a shell that's connected to your terminal using the `-i` and `-t`
|
||||
arguments to `kubectl exec`, for example:
|
||||
|
||||
```shell
|
||||
kubectl exec -it cassandra -- sh
|
||||
```
|
||||
|
||||
For more details, see [Get a Shell to a Running Container](
|
||||
/docs/tasks/debug-application-cluster/get-shell-running-container/).
|
||||
|
||||
## Debugging with an ephemeral debug container {#ephemeral-container}
|
||||
|
||||
{{< feature-state state="alpha" for_k8s_version="v1.18" >}}
|
||||
|
||||
{{< glossary_tooltip text="Ephemeral containers" term_id="ephemeral-container" >}}
|
||||
are useful for interactive troubleshooting when `kubectl exec` is insufficient
|
||||
because a container has crashed or a container image doesn't include debugging
|
||||
utilities, such as with [distroless images](
|
||||
https://github.com/GoogleContainerTools/distroless). `kubectl` has an alpha
|
||||
command that can create ephemeral containers for debugging beginning with version
|
||||
`v1.18`.
|
||||
|
||||
### Example debugging using ephemeral containers {#ephemeral-container-example}
|
||||
|
||||
{{< note >}}
|
||||
The examples in this section require the `EphemeralContainers` [feature gate](
|
||||
/docs/reference/command-line-tools-reference/feature-gates/) enabled in your
|
||||
cluster and `kubectl` version v1.18 or later.
|
||||
{{< /note >}}
|
||||
|
||||
You can use the `kubectl alpha debug` command to add ephemeral containers to a
|
||||
running Pod. First, create a pod for the example:
|
||||
|
||||
```shell
|
||||
kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
This section use the `pause` container image in examples because it does not
|
||||
contain userland debugging utilities, but this method works with all container
|
||||
images.
|
||||
{{< /note >}}
|
||||
|
||||
If you attempt to use `kubectl exec` to create a shell you will see an error
|
||||
because there is no shell in this container image.
|
||||
|
||||
```shell
|
||||
kubectl exec -it pause -- sh
|
||||
```
|
||||
|
||||
```
|
||||
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
|
||||
```
|
||||
|
||||
You can instead add a debugging container using `kubectl alpha debug`. If you
|
||||
specify the `-i`/`--interactive` argument, `kubectl` will automatically attach
|
||||
to the console of the Ephemeral Container.
|
||||
|
||||
```shell
|
||||
kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
|
||||
```
|
||||
|
||||
```
|
||||
Defaulting debug container name to debugger-8xzrl.
|
||||
If you don't see a command prompt, try pressing enter.
|
||||
/ #
|
||||
```
|
||||
|
||||
This command adds a new busybox container and attaches to it. The `--target`
|
||||
parameter targets the process namespace of another container. It's necessary
|
||||
here because `kubectl run` does not enable [process namespace sharing](
|
||||
/docs/tasks/configure-pod-container/share-process-namespace/) in the pod it
|
||||
creates.
|
||||
|
||||
{{< note >}}
|
||||
The `--target` parameter must be supported by the {{< glossary_tooltip
|
||||
text="Container Runtime" term_id="container-runtime" >}}. When not supported,
|
||||
the Ephemeral Container may not be started, or it may be started with an
|
||||
isolated process namespace.
|
||||
{{< /note >}}
|
||||
|
||||
You can view the state of the newly created ephemeral container using `kubectl describe`:
|
||||
|
||||
```shell
|
||||
kubectl describe pod ephemeral-demo
|
||||
```
|
||||
|
||||
```
|
||||
...
|
||||
Ephemeral Containers:
|
||||
debugger-8xzrl:
|
||||
Container ID: docker://b888f9adfd15bd5739fefaa39e1df4dd3c617b9902082b1cfdc29c4028ffb2eb
|
||||
Image: busybox
|
||||
Image ID: docker-pullable://busybox@sha256:1828edd60c5efd34b2bf5dd3282ec0cc04d47b2ff9caa0b6d4f07a21d1c08084
|
||||
Port: <none>
|
||||
Host Port: <none>
|
||||
State: Running
|
||||
Started: Wed, 12 Feb 2020 14:25:42 +0100
|
||||
Ready: False
|
||||
Restart Count: 0
|
||||
Environment: <none>
|
||||
Mounts: <none>
|
||||
...
|
||||
```
|
||||
|
||||
Use `kubectl delete` to remove the Pod when you're finished:
|
||||
|
||||
```shell
|
||||
kubectl delete pod ephemeral-demo
|
||||
```
|
||||
|
||||
<!--
|
||||
Planned future sections include:
|
||||
|
||||
* Debugging with a copy of the pod
|
||||
|
||||
See https://git.k8s.io/enhancements/keps/sig-cli/20190805-kubectl-debug.md
|
||||
-->
|
||||
|
||||
## Debugging via a shell on the node {#node-shell-session}
|
||||
|
||||
If none of these approaches work, you can find the host machine that the pod is
|
||||
running on and SSH into that host, but this should generally not be necessary
|
||||
given tools in the Kubernetes API. Therefore, if you find yourself needing to
|
||||
ssh into a machine, please file a feature request on GitHub describing your use
|
||||
case and why these tools are insufficient.
|
||||
|
||||
{{% /capture %}}
|
||||
Loading…
Reference in New Issue