kubernetes
/
website
mirror of https://github.com/kubernetes/website.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #41754 from pacoxu/Forensic-container-analysis 

add Chinese translate for Forensic container checkpointing
pull/41765/head
Kubernetes Prow Robot 2023-06-25 20:57:07 -07:00 committed by GitHub
parent ce91c8e3ee 5d9137280e
commit 04ba19ca84
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 983 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

425
content/zh-cn/blog/_posts/2022-12-05-forensic-container-checkpointing/index.md Normal file
View File

@ -0,0 +1,425 @@
---
layout: blog
title: "Kubernetes 的取证容器检查点"
date: 2022-12-05
slug: forensic-container-checkpointing-alpha
---
**作者:** [Adrian Reber](https://github.com/adrianreber) (Red Hat)
<!--
**Authors:** Adrian Reber (Red Hat)
-->
<!--
Forensic container checkpointing is based on [Checkpoint/Restore In
Userspace](https://criu.org/) (CRIU) and allows the creation of stateful copies
of a running container without the container knowing that it is being
checkpointed. The copy of the container can be analyzed and restored in a
sandbox environment multiple times without the original container being aware
of it. Forensic container checkpointing was introduced as an alpha feature in
Kubernetes v1.25.
-->
取证容器检查点（Forensic container checkpointing）基于 [CRIU][criu]（Checkpoint/Restore In Userspace ，用户空间的检查点/恢复），
并允许创建正在运行的容器的有状态副本，而容器不知道它正在被检查。容器的副本，可以在沙箱环境中被多次分析和恢复，而原始容器并不知道。
取证容器检查点是作为一个 alpha 特性在 Kubernetes v1.25 中引入的。
<!--
## How does it work?
-->
## 工作原理
<!--
With the help of CRIU it is possible to checkpoint and restore containers.
CRIU is integrated in runc, crun, CRI-O and containerd and forensic container
checkpointing as implemented in Kubernetes uses these existing CRIU
integrations.
-->
在 CRIU 的帮助下，检查（checkpoint）和恢复容器成为可能。CRIU 集成在 runc、crun、CRI-O 和 containerd 中，
而在 Kubernetes 中实现的取证容器检查点使用这些现有的 CRIU 集成。
<!--
## Why is it important?
-->
## 这一特性为何重要？
<!--
With the help of CRIU and the corresponding integrations it is possible to get
all information and state about a running container on disk for later forensic
analysis. Forensic analysis might be important to inspect a suspicious
container without stopping or influencing it. If the container is really under
attack, the attacker might detect attempts to inspect the container. Taking a
checkpoint and analysing the container in a sandboxed environment offers the
possibility to inspect the container without the original container and maybe
attacker being aware of the inspection.
-->
借助 CRIU 和相应的集成，可以获得磁盘上正在运行的容器的所有信息和状态，供以后进行取证分析。
取证分析对于在不阻止或影响可疑容器的情况下，对其进行检查可能很重要。如果容器确实受到攻击，攻击者可能会检测到检查容器的企图。
获取检查点并在沙箱环境中分析容器，提供了在原始容器和可能的攻击者不知道检查的情况下检查容器的可能性。
<!--
In addition to the forensic container checkpointing use case, it is also
possible to migrate a container from one node to another node without loosing
the internal state. Especially for stateful containers with long initialization
times restoring from a checkpoint might save time after a reboot or enable much
faster startup times.
-->
除了取证容器检查点用例，还可以在不丢失内部状态的情况下，将容器从一个节点迁移到另一个节点。
特别是对于初始化时间长的有状态容器，从检查点恢复，可能会节省重新启动后的时间，或者实现更快的启动时间。
<!--
## How do I use container checkpointing?
-->
## 如何使用容器检查点？
<!--
The feature is behind a [feature gate][container-checkpoint-feature-gate], so
make sure to enable the `ContainerCheckpoint` gate before you can use the new
feature.
-->
该功能在[特性门控][container-checkpoint-feature-gate]后面，因此在使用这个新功能之前，
请确保启用了 ContainerCheckpoint 特性门控。
<!--
The runtime must also support container checkpointing:
* containerd: support is currently under discussion. See containerd
pull request [#6965][containerd-checkpoint-restore-pr] for more details.
* CRI-O: v1.25 has support for forensic container checkpointing.
-->
运行时还必须支持容器检查点：
* containerd：相关支持目前正在讨论中。有关更多详细信息，请参见 [containerd pull request #6965][containerd-checkpoint-restore-pr]。
* CRI-O：v1.25 支持取证容器检查点。
<!--
### Usage example with CRI-O
-->
## CRI-O 的使用示例
<!--
To use forensic container checkpointing in combination with CRI-O, the runtime
needs to be started with the command-line option `--enable-criu-support=true`.
For Kubernetes, you need to run your cluster with the `ContainerCheckpoint`
feature gate enabled. As the checkpointing functionality is provided by CRIU it
is also necessary to install CRIU. Usually runc or crun depend on CRIU and
therefore it is installed automatically.
-->
要将取证容器检查点与 CRI-O 结合使用，需要使用命令行选项--enable-criu-support=true 启动运行时。
Kubernetes 方面，你需要在启用 ContainerCheckpoint 特性门控的情况下运行你的集群。
由于检查点功能是由 CRIU 提供的，因此也有必要安装 CRIU。
通常 runc 或 crun 依赖于 CRIU，因此它是自动安装的。
<!--
It is also important to mention that at the time of writing the checkpointing functionality is
to be considered as an alpha level feature in CRI-O and Kubernetes and the
security implications are still under consideration.
-->
值得一提的是，在编写本文时，检查点功能被认为是 CRI-O 和 Kubernetes 中的一个 alpha 级特性，其安全影响仍在评估之中。
<!--
Once containers and pods are running it is possible to create a checkpoint.
[Checkpointing](https://kubernetes.io/docs/reference/node/kubelet-checkpoint-api/)
is currently only exposed on the **kubelet** level. To checkpoint a container,
you can run `curl` on the node where that container is running, and trigger a
checkpoint:
```shell
curl -X POST "https://localhost:10250/checkpoint/namespace/podId/container"
```
-->
一旦容器和 pod 开始运行，就可以创建一个检查点。[检查点][kubelet-checkpoint-api]目前只在 **kubelet** 级别暴露。
要检查一个容器，可以在运行该容器的节点上运行 curl，并触发一个检查点：
```shell
curl -X POST "https://localhost:10250/checkpoint/namespace/podId/container"
```
<!--
For a container named *counter* in a pod named *counters* in a namespace named
*default* the *kubelet* API endpoint is reachable at:
```shell
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
```
-->
对于 **default** 命名空间中 **counters** Pod 中名为 **counter** 的容器，可通过以下方式访问 **kubelet** API 端点：
```shell
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
```
<!--
For completeness the following `curl` command-line options are necessary to
have `curl` accept the *kubelet*'s self signed certificate and authorize the
use of the *kubelet* `checkpoint` API:
```shell
--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key
```
-->
为了完整起见，以下 `curl` 命令行选项对于让 `curl` 接受 **kubelet** 的自签名证书并授权使用
**kubelet** 检查点 API 是必要的：
```shell
--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key
```
<!--
Triggering this **kubelet** API will request the creation of a checkpoint from
CRI-O. CRI-O requests a checkpoint from your low-level runtime (for example,
`runc`). Seeing that request, `runc` invokes the `criu` tool
to do the actual checkpointing.
Once the checkpointing has finished the checkpoint should be available at
`/var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar`
You could then use that tar archive to restore the container somewhere else.
-->
触发这个 **kubelet** API 将从 CRI-O 请求创建一个检查点，CRI-O 从你的低级运行时（例如 `runc`）请求一个检查点。
看到这个请求，`runc` 调用 `criu` 工具来执行实际的检查点操作。
检查点操作完成后，检查点应该位于
`/var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar`
然后，你可以使用 tar 归档文件在其他地方恢复容器。
<!--
### Restore a checkpointed container outside of Kubernetes (with CRI-O) {#restore-checkpointed-container-standalone}
-->
### 在 Kubernetes 外恢复检查点容器（使用 CRI-O）
<!--
With the checkpoint tar archive it is possible to restore the container outside
of Kubernetes in a sandboxed instance of CRI-O. For better user experience
during restore, I recommend that you use the latest version of CRI-O from the
*main* CRI-O GitHub branch. If you're using CRI-O v1.25, you'll need to
manually create certain directories Kubernetes would create before starting the
container.
-->
使用检查点 tar 归档文件，可以在 Kubernetes 之外的 CRI-O 沙箱实例中恢复容器。
为了在恢复过程中获得更好的用户体验，建议你使用 CRI-O GitHub 的 **main** 分支中最新版本的 CRI-O。
如果你使用的是 CRI-O v1.25，你需要在启动容器之前手动创建 Kubernetes 会创建的某些目录。
<!--
The first step to restore a container outside of Kubernetes is to create a pod sandbox
using *crictl*:
```shell
crictl runp pod-config.json
```
-->
在 Kubernetes 外恢复容器的第一步是使用 **crictl** 创建一个 pod 沙箱：
```shell
crictl runp pod-config.json
```
<!--
Then you can restore the previously checkpointed container into the newly created pod sandbox:
```shell
crictl create <POD_ID> container-config.json pod-config.json
```
-->
然后，你可以将之前的检查点容器恢复到新创建的 pod 沙箱中：
```shell
crictl create <POD_ID> container-config.json pod-config.json
```
<!--
Instead of specifying a container image in a registry in `container-config.json`
you need to specify the path to the checkpoint archive that you created earlier:
```json
{
"metadata": {
"name": "counter"
},
"image":{
"image": "/var/lib/kubelet/checkpoints/<checkpoint-archive>.tar"
}
}
```
-->
你不需要在 container-config.json 的注册表中指定容器镜像，而是需要指定你之前创建的检查点归档文件的路径：
```json
{
"metadata": {
"name": "counter"
},
"image":{
"image": "/var/lib/kubelet/checkpoints/<checkpoint-archive>.tar"
}
}
```
<!--
Next, run `crictl start <CONTAINER_ID>` to start that container, and then a
copy of the previously checkpointed container should be running.
-->
接下来，运行 crictl start <CONTAINER_ID>来启动该容器，然后应该会运行先前检查点容器的副本。
<!--
### Restore a checkpointed container within of Kubernetes {#restore-checkpointed-container-k8s}
-->
### 在 Kubernetes 中恢复检查点容器
<!--
To restore the previously checkpointed container directly in Kubernetes it is
necessary to convert the checkpoint archive into an image that can be pushed to
a registry.
-->
要在 Kubernetes 中直接恢复之前的检查点容器，需要将检查点归档文件转换成可以推送到注册中心的镜像。
<!--
One possible way to convert the local checkpoint archive consists of the
following steps with the help of [buildah](https://buildah.io/):
```shell
newcontainer=$(buildah from scratch)
buildah add $newcontainer /var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar /
buildah config --annotation=io.kubernetes.cri-o.annotations.checkpoint.name=<container-name> $newcontainer
buildah commit $newcontainer checkpoint-image:latest
buildah rm $newcontainer
```
-->
转换本地检查点存档的一种方法包括在 [buildah][buildah] 的帮助下执行以下步骤：
```shell
newcontainer=$(buildah from scratch)
buildah add $newcontainer /var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar /
buildah config --annotation=io.kubernetes.cri-o.annotations.checkpoint.name=<container-name> $newcontainer
buildah commit $newcontainer checkpoint-image:latest
buildah rm $newcontainer
```
<!--
The resulting image is not standardized and only works in combination with
CRI-O. Please consider this image format as pre-alpha. There are ongoing
[discussions][image-spec-discussion] to standardize the format of checkpoint
images like this. Important to remember is that this not yet standardized image
format only works if CRI-O has been started with `--enable-criu-support=true`.
The security implications of starting CRI-O with CRIU support are not yet clear
and therefore the functionality as well as the image format should be used with
care.
-->
生成的镜像未经标准化，只能与 CRI-O 结合使用。请将此镜像格式视为 pre-alpha 格式。
社区正在[讨论][image-spec-discussion]如何标准化这样的检查点镜像格式。
重要的是要记住，这种尚未标准化的镜像格式只有在 CRI-O 已经用`--enable-criu-support=true` 启动时才有效。
在 CRIU 支持下启动 CRI-O 的安全影响尚不清楚，因此应谨慎使用功能和镜像格式。
<!--
Now, you'll need to push that image to a container image registry. For example:
```shell
buildah push localhost/checkpoint-image:latest container-image-registry.example/user/checkpoint-image:latest
```
-->
现在，你需要将该镜像推送到容器镜像注册中心。例如：
```shell
buildah push localhost/checkpoint-image:latest container-image-registry.example/user/checkpoint-image:latest
```
<!--
To restore this checkpoint image (`container-image-registry.example/user/checkpoint-image:latest`), the
image needs to be listed in the specification for a Pod. Here's an example
manifest:
```yaml
apiVersion: v1
kind: Pod
metadata:
namePrefix: example-
spec:
containers:
- name: <container-name>
image: container-image-registry.example/user/checkpoint-image:latest
nodeName: <destination-node>
```
-->
要恢复此检查点镜像（container-image-registry.example/user/checkpoint-image:latest），
该镜像需要在 Pod 的规约中列出。下面是一个清单示例：
```yaml
apiVersion: v1
kind: Pod
metadata:
namePrefix: example-
spec:
containers:
- name: <container-name>
image: container-image-registry.example/user/checkpoint-image:latest
nodeName: <destination-node>
```
<!--
Kubernetes schedules the new Pod onto a node. The kubelet on that node
instructs the container runtime (CRI-O in this example) to create and start a
container based on an image specified as `registry/user/checkpoint-image:latest`.
CRI-O detects that `registry/user/checkpoint-image:latest`
is a reference to checkpoint data rather than a container image. Then,
instead of the usual steps to create and start a container,
CRI-O fetches the checkpoint data and restores the container from that
specified checkpoint.
-->
Kubernetes 将新的 Pod 调度到一个节点上。该节点上的 kubelet 指示容器运行时（本例中为 CRI-O）
基于指定为 `registry/user/checkpoint-image:latest` 的镜像创建并启动容器。
CRI-O 检测到 `registry/user/checkpoint-image:latest` 是对检查点数据的引用，而不是容器镜像。
然后，与创建和启动容器的通常步骤不同，CRI-O 获取检查点数据，并从指定的检查点恢复容器。
<!--
The application in that Pod would continue running as if the checkpoint had not been taken;
within the container, the application looks and behaves like any other container that had been
started normally and not restored from a checkpoint.
-->
该 Pod 中的应用程序将继续运行，就像检查点未被获取一样；在该容器中，
应用程序的外观和行为，与正常启动且未从检查点恢复的任何其他容器相似。
<!--
With these steps, it is possible to replace a Pod running on one node
with a new equivalent Pod that is running on a different node,
and without losing the state of the containers in that Pod.
-->
通过这些步骤，可以用在不同节点上运行的新的等效 Pod，替换在一个节点上运行的 Pod，而不会丢失该 Pod中容器的状态。
<!--
## How do I get involved?
-->
## 如何参与？
<!--
You can reach SIG Node by several means:
* Slack: [#sig-node](https://kubernetes.slack.com/messages/sig-node)
* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-node)
-->
你可以通过多种方式参与 SIG Node：
* Slack: [#sig-node][sig-node]
* [Mailing list][Mailing list]
<!--
## Further reading
-->
## 延伸阅读
<!--
Please see the follow-up article [Forensic container
analysis][forensic-container-analysis] for details on how a container checkpoint
can be analyzed.
-->
有关如何分析容器检查点的详细信息，请参阅后续文章[取证容器分析][forensic-container-analysis]。
[forensic-container-analysis]: /zh-cn/blog/2023/03/10/forensic-container-analysis/
[criu]: https://criu.org/
[containerd-checkpoint-restore-pr]: https://github.com/containerd/containerd/pull/6965
[container-checkpoint-feature-gate]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
[image-spec-discussion]: <https://github.com/opencontainers/image-spec/issues/962>
[kubelet-checkpoint-api]: <https://kubernetes.io/docs/reference/node/kubelet-checkpoint-api/>
[buildah]: <https://buildah.io/>
[sig-node]: <https://kubernetes.slack.com/messages/sig-node>
[Mailing list]: <https://groups.google.com/forum/#!forum/kubernetes-sig-node>

558
content/zh-cn/blog/_posts/2023-03-10-forensic-container-analysis/index.md Normal file
View File

@ -0,0 +1,558 @@
---
layout: blog
title: "Kubernetes 的容器检查点分析"
date: 2023-03-10
slug: forensic-container-analysis
---
<!--
layout: blog
title: "Forensic container analysis"
date: 2023-03-10
slug: forensic-container-analysis
-->
**作者:** [Adrian Reber](https://github.com/adrianreber) (Red Hat)
<!--
**Authors:** Adrian Reber (Red Hat)
-->
**译者**：[Paco Xu](https://github.com/pacoxu) (Daocloud)
<!--
In my previous article, [Forensic container checkpointing in
Kubernetes][forensic-blog], I introduced checkpointing in Kubernetes
and how it has to be setup and how it can be used. The name of the
feature is Forensic container checkpointing, but I did not go into
any details how to do the actual analysis of the checkpoint created by
Kubernetes. In this article I want to provide details how the
checkpoint can be analyzed.
-->
在我之前的文章 [Kubernetes 中的取证容器检查点][forensic-blog] 中，我介绍了检查点以及如何创建和使用它。
该特性的名称是取证容器检查点，但我没有详细介绍如何对 Kubernetes 创建的检查点进行实际分析。
在本文中，我想提供如何分析检查点的详细信息。
<!--
Checkpointing is still an alpha feature in Kubernetes and this article
wants to provide a preview how the feature might work in the future.
-->
检查点仍然是 Kubernetes 中的一个 alpha 功能，本文希望提供该功能未来如何工作的预览。
<!--
## Preparation
-->
## 准备
<!--
Details about how to configure Kubernetes and the underlying CRI implementation
to enable checkpointing support can be found in my [Forensic container
checkpointing in Kubernetes][forensic-blog] article.
As an example I prepared a container image (`quay.io/adrianreber/counter:blog`)
which I want to checkpoint and then analyze in this article. This container allows
me to create files in the container and also store information in memory which
I later want to find in the checkpoint.
-->
有关如何配置 Kubernetes 和底层 CRI 实现以启用检查点支持的详细信息，请参阅 [Kubernetes 中的取证容器检查点][forensic-blog]文章。
作为示例，我准备了一个容器镜像（`quay.io/adrianreber/counter:blog`），我想对其进行检查点，然后在本文中进行分析。
这个容器允许我在容器中创建文件，并将信息存储在内存中，稍后我想在检查点中找到这些信息。
<!--
To run that container I need a pod, and for this example I am using the following Pod manifest:
-->
要运行该容器，我需要一个 pod，在本示例中，我使用以下 Pod 清单：
```yaml
apiVersion: v1
kind: Pod
metadata:
name: counters
spec:
containers:
- name: counter
image: quay.io/adrianreber/counter:blog
```
<!--
This results in a container called `counter` running in a pod called `counters`.
Once the container is running I am performing following actions with that
container:
-->
这会导致一个名为 `counter` 的容器在名为 `counters` 的 Pod 中运行。
容器运行后，我将对该容器执行以下操作：
```console
$ kubectl get pod counters --template '{{.status.podIP}}'
10.88.0.25
$ curl 10.88.0.25:8088/create?test-file
$ curl 10.88.0.25:8088/secret?RANDOM_1432_KEY
$ curl 10.88.0.25:8088
```
<!--
The first access creates a file called `test-file` with the content `test-file`
in the container and the second access stores my secret information
(`RANDOM_1432_KEY`) somewhere in the container's memory. The last access just
adds an additional line to the internal log file.
The last step before I can analyze the checkpoint it to tell Kubernetes to create
the checkpoint. As described in the previous article this requires access to the
*kubelet* only `checkpoint` API endpoint.
For a container named *counter* in a pod named *counters* in a namespace named
*default* the *kubelet* API endpoint is reachable at:
-->
1. 第一次访问在容器中创建一个名为 `test-file` 的文件，其内容为 `test-file`;
2. 第二次访问将我的秘密信息（`RANDOM_1432_KEY`）存储在容器内存中的某处;
3. 最后一次访问在内部日志文件中添加了一行。
在分析检查点之前的最后一步是告诉 Kubernetes 创建检查点。如上一篇文章所述，这需要访问 **kubelet** 唯一的“检查点” API 端点。
对于 **default** 命名空间中 **counters** Pod 中名为 **counter** 的容器，
可通过以下方式访问 **kubelet** API 端点：
<!--
```shell
# run this on the node where that Pod is executing
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
```
-->
```shell
# 在运行 Pod 的节点上运行这条命令
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
```
<!--
For completeness the following `curl` command-line options are necessary to
have `curl` accept the *kubelet*'s self signed certificate and authorize the
use of the *kubelet* `checkpoint` API:
-->
为了完整起见，以下 curl 命令行选项对于让 curl 接受 **kubelet** 的自签名证书并授权使用 **kubelet** 检查点 API 是必要的：
```shell
--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key
```
<!--
Once the checkpointing has finished the checkpoint should be available at
`/var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar`
-->
检查点操作完成后，检查点应该位于 `/var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar`
<!--
In the following steps of this article I will use the name `checkpoint.tar`
when analyzing the checkpoint archive.
-->
在本文的以下步骤中，我将在分析检查点归档时使用名称 `checkpoint.tar`。
<!--
## Checkpoint archive analysis using `checkpointctl`
-->
## 使用 `checkpointctl` 进行检查点归档分析
<!--
To get some initial information about the checkpointed container I am using the
tool [checkpointctl][checkpointctl] like this:
-->
我使用工具 [checkpointctl][checkpointctl] 获取有关检查点容器的一些初始信息，如下所示：
```console
$ checkpointctl show checkpoint.tar --print-stats
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
| CONTAINER | IMAGE | ID | RUNTIME | CREATED | ENGINE | IP | CHKPT SIZE | ROOT FS DIFF SIZE |
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
| counter | quay.io/adrianreber/counter:blog | 059a219a22e5 | runc | 2023-03-02T06:06:49 | CRI-O | 10.88.0.23 | 8.6 MiB | 3.0 KiB |
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
CRIU dump statistics
+---------------+-------------+--------------+---------------+---------------+---------------+
| FREEZING TIME | FROZEN TIME | MEMDUMP TIME | MEMWRITE TIME | PAGES SCANNED | PAGES WRITTEN |
+---------------+-------------+--------------+---------------+---------------+---------------+
| 100809 us | 119627 us | 11602 us | 7379 us | 7800 | 2198 |
+---------------+-------------+--------------+---------------+---------------+---------------+
```
<!--
This gives me already some information about the checkpoint in that checkpoint
archive. I can see the name of the container, information about the container
runtime and container engine. It also lists the size of the checkpoint (`CHKPT
SIZE`). This is mainly the size of the memory pages included in the checkpoint,
but there is also information about the size of all changed files in the
container (`ROOT FS DIFF SIZE`).
-->
这展示了有关该检查点归档中的检查点的一些信息。我们可以看到容器的名称、有关容器运行时和容器引擎的信息。
它还列出了检查点的大小（`CHKPT SIZE`）。
这主要是检查点中包含的内存页的大小，同时也有有关容器中所有更改文件的大小的信息（`ROOT FS DIFF SIZE`）。
<!--
The additional parameter `--print-stats` decodes information in the checkpoint
archive and displays them in the second table (*CRIU dump statistics*). This
information is collected during checkpoint creation and gives an overview how much
time CRIU needed to checkpoint the processes in the container and how many
memory pages were analyzed and written during checkpoint creation.
-->
使用附加参数 `--print-stats` 可以解码检查点归档中的信息并将其显示在第二个表中（**CRIU 转储统计信息**）。
此信息是在检查点创建期间收集的，并概述了 CRIU 对容器中的进程生成检查点所需的时间以及在检查点创建期间分析和写入了多少内存页。
<!--
## Digging deeper
-->
## 深入挖掘
<!--
With the help of `checkpointctl` I am able to get some high level information
about the checkpoint archive. To be able to analyze the checkpoint archive
further I have to extract it. The checkpoint archive is a *tar* archive and can
be extracted with the help of `tar xf checkpoint.tar`.
Extracting the checkpoint archive will result in following files and directories:
* `bind.mounts` - this file contains information about bind mounts and is needed
during restore to mount all external files and directories at the right location
* `checkpoint/` - this directory contains the actual checkpoint as created by
CRIU
* `config.dump` and `spec.dump` - these files contain metadata about the container
which is needed during restore
* `dump.log` - this file contains the debug output of CRIU created during
checkpointing
* `stats-dump` - this file contains the data which is used by `checkpointctl`
to display dump statistics (`--print-stats`)
* `rootfs-diff.tar` - this file contains all changed files on the container's
file-system
-->
借助 `checkpointctl`，我可以获得有关检查点归档的一些高级信息。为了能够进一步分析检查点归档，我必须将其提取。
检查点归档是 **tar** 归档文件，可以借助 `tar xf checkpoint.tar` 进行解压。
展开检查点存档时，将创建以下文件和目录：
* `bind.mounts` - 该文件包含有关绑定挂载的信息，并且需要在恢复期间需要将所有外部文件和目录挂载到正确的位置。
* `checkpoint/` - 该目录包含 CRIU 创建的实际检查点，
* `config.dump` 和 `spec.dump` - 这些文件包含恢复期间所需的有关容器的元数据。
* `dump.log` - 该文件包含在检查点期间创建的 CRIU 的调试输出。
* `stats-dump` - 此文件包含 `checkpointctl` 用于通过 `--print-stats` 显示转储统计信息的数据。
* `rootfs-diff.tar` - 该文件包含容器文件系统上所有已更改的文件。
<!--
### File-system changes - `rootfs-diff.tar`
-->
### 更改文件系统 - `rootfs-diff.tar`
<!--
The first step to analyze the container's checkpoint further is to look at
the files that have changed in my container. This can be done by looking at the
file `rootfs-diff.tar`:
-->
进一步分析容器检查点的第一步是查看容器内已更改的文件。这可以通过引用 `rootfs-diff.tar` 文件来完成。
```console
$ tar xvf rootfs-diff.tar
home/counter/logfile
home/counter/test-file
```
<!--
Now the files that changed in the container can be studied:
-->
现在你可以检查容器内已更改的文件。
```console
$ cat home/counter/logfile
10.88.0.1 - - [02/Mar/2023 06:07:29] "GET /create?test-file HTTP/1.1" 200 -
10.88.0.1 - - [02/Mar/2023 06:07:40] "GET /secret?RANDOM_1432_KEY HTTP/1.1" 200 -
10.88.0.1 - - [02/Mar/2023 06:07:43] "GET / HTTP/1.1" 200 -
$ cat home/counter/test-file
test-file 
```
<!--
Compared to the container image (`quay.io/adrianreber/counter:blog`) this
container is based on, I can see that the file `logfile` contains information
about all access to the service the container provides and the file `test-file`
was created just as expected.
With the help of `rootfs-diff.tar` it is possible to inspect all files that
were created or changed compared to the base image of the container.
-->
与该容器所基于的容器镜像（`quay.io/adrianreber/counter:blog`）相比，
它包含容器提供的服务的所有访问信息以及预期创建的 `logfile` 可以检查 `test-file` 文件。
在 `rootfs-diff.tar` 的帮助下，可以根据容器的基本镜像检查所有创建或修改的文件。
<!--
### Analyzing the checkpointed processes - `checkpoint/`
-->
### 分析检查点进程 - `checkpoint/`
<!--
The directory `checkpoint/` contains data created by CRIU while checkpointing
the processes in the container. The content in the directory `checkpoint/`
consists of different [image files][image-files] which can be analyzed with the
help of the tool [CRIT][crit] which is distributed as part of CRIU.
First lets get an overview of the processes inside of the container:
-->
目录 `checkpoint/` 包含 CRIU 在容器内对进程进行检查点时创建的数据。
目录 `checkpoint/` 的内容由各种[镜像文件][image-files] 组成，可以使用作为 CRIU 一部分分发的 [CRIT][crit] 工具进行分析。
首先，我们先了解一下容器的内部流程。
```console
$ crit show checkpoint/pstree.img | jq .entries[].pid
1
7
8
```
<!--
This output means that I have three processes inside of the container's PID
namespace with the PIDs: 1, 7, 8
This is only the view from the inside of the container's PID namespace. During
restore exactly these PIDs will be recreated. From the outside of the
container's PID namespace the PIDs will change after restore.
The next step is to get some additional information about these three processes:
-->
此输出意味着容器的 PID 命名空间内有 3 个进程（PID 为 1、7 和 8）。
这只是容器 PID 命名空间的内部视图。这些 PID 在恢复过程中会准确地重新创建。从容器的 PID 命名空间外部，PID 将在恢复后更改。
下一步是获取有关这三个进程的更多信息。
```console
$ crit show checkpoint/core-1.img | jq .entries[0].tc.comm
"bash"
$ crit show checkpoint/core-7.img | jq .entries[0].tc.comm
"counter.py"
$ crit show checkpoint/core-8.img | jq .entries[0].tc.comm
"tee"
```
<!--
This means the three processes in my container are `bash`, `counter.py` (a Python
interpreter) and `tee`. For details about the parent child relations of these processes there
is more data to be analyzed in `checkpoint/pstree.img`.
Let's compare the so far collected information to the still running container:
-->
这意味着容器内的三个进程是 `bash`、`counter.py`（Python 解释器）和 `tee`。
`checkpoint/pstree.img` 中有更多数据可供分析，以获取有关进程起源的详细信息。
让我们将目前为止收集到的信息与仍在运行的容器进行比较。
```console
$ crictl inspect --output go-template --template "{{(index .info.pid)}}" 059a219a22e56
722520
$ ps auxf | grep -A 2 722520
fedora 722520 \_ bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile
fedora 722541 \_ /usr/bin/python3 /home/counter/counter.py
fedora 722542 \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /home/counter/logfile
$ cat /proc/722520/comm
bash
$ cat /proc/722541/comm
counter.py
$ cat /proc/722542/comm
tee
```
<!--
In this output I am first retrieving the PID of the first process in the
container and then I am looking for that PID and child processes on the system
where the container is running. I am seeing three processes and the first one is
"bash" which is PID 1 inside of the containers PID namespace. Then I am looking
at `/proc/<PID>/comm` and I can find the exact same value
as in the checkpoint image.
Important to remember is that the checkpoint will contain the view from within the
container's PID namespace because that information is important to restore the
processes.
One last example of what `crit` can tell us about the container is the information
about the UTS namespace:
-->
在此输出中，我们首先获取容器中第一个进程的 PID。在运行容器的系统上，它会查找其 PID 和子进程。
你应该看到三个进程，第一个进程是 `bash`，容器 PID 命名空间中的 PID 为 1。
然后查看 `/proc/<PID>/comm`，可以找到与检查点镜像完全相同的值。
需要记住的重点是，检查点包含容器的 PID 命名空间内的视图。因为这些信息对于恢复进程非常重要。
`crit` 告诉我们有关容器的最后一个例子是有关 UTS 命名空间的信息。
```console
$ crit show checkpoint/utsns-12.img
{
"magic": "UTSNS",
"entries": [
{
"nodename": "counters",
"domainname": "(none)"
}
]
}
```
<!--
This tells me that the hostname inside of the UTS namespace is `counters`.
For every resource CRIU collected during checkpointing the `checkpoint/`
directory contains corresponding image files which can be analyzed with the help
of `crit`.
-->
这里输出表示 UTS 命名空间中的主机名是 `counters`。
对于检查点创建期间收集的每个资源 CRIU，`checkpoint/` 目录包含相应的镜像文件。可以使用 `crit` 来分析该镜像文件。
<!--
#### Looking at the memory pages
-->
#### 查看内存页面
<!--
In addition to the information from CRIU that can be decoded with the help
of CRIT, there are also files containing the raw memory pages written by
CRIU to disk:
-->
除了可以借助 CRIT 解码的 CRIU 信息之外，还有包含 CRIU 写入磁盘的原始内存页的文件：
```console
$ ls checkpoint/pages-*
checkpoint/pages-1.img checkpoint/pages-2.img checkpoint/pages-3.img
```
<!--
When I initially used the container I stored a random key (`RANDOM_1432_KEY`)
somewhere in the memory. Let see if I can find it:
-->
当我最初使用该容器时，我在内存中的某个位置存储了一个随机密钥。让我看看是否能找到它：
```console
$ grep -ao RANDOM_1432_KEY checkpoint/pages-*
checkpoint/pages-2.img:RANDOM_1432_KEY
```
<!--
And indeed, there is my data. This way I can easily look at the content
of all memory pages of the processes in the container, but it is also
important to remember that anyone that can access the checkpoint
archive has access to all information that was stored in the memory of the
container's processes.
-->
确实有我的数据。通过这种方式，我可以轻松查看容器中进程的所有内存页面的内容，
但需要注意的是可以访问检查点存档的任何人都可以访问存储在容器进程内存中的所有信息。
<!--
#### Using gdb for further analysis
-->
#### 使用 gdb 进行进一步分析
<!--
Another possibility to look at the checkpoint images is `gdb`. The CRIU repository
contains the script [coredump][criu-coredump] which can convert a checkpoint
into a coredump file:
-->
查看检查点镜像的另一种方法是 `gdb`。CRIU 存储库包含脚本 [coredump][criu-coredump]，它可以将检查点转换为 coredump 文件：
```console
$ /home/criu/coredump/coredump-python3
$ ls -al core*
core.1 core.7 core.8
```
<!--
Running the `coredump-python3` script will convert the checkpoint images into
one coredump file for each process in the container. Using `gdb` I can also look
at the details of the processes:
-->
运行 `coredump-python3` 脚本会将检查点镜像转换为容器中每个进程一个的 coredump 文件。 使用 `gdb` 我还可以查看进程的详细信息：
```console
$ echo info registers | gdb --core checkpoint/core.1 -q
[New LWP 1]
Core was generated by `bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile'.
#0 0x00007fefba110198 in ?? ()
(gdb)
rax 0x3d 61
rbx 0x8 8
rcx 0x7fefba11019a 140667595587994
rdx 0x0 0
rsi 0x7fffed9c1110 140737179816208
rdi 0xffffffff 4294967295
rbp 0x1 0x1
rsp 0x7fffed9c10e8 0x7fffed9c10e8
r8 0x1 1
r9 0x0 0
r10 0x0 0
r11 0x246 582
r12 0x0 0
r13 0x7fffed9c1170 140737179816304
r14 0x0 0
r15 0x0 0
rip 0x7fefba110198 0x7fefba110198
eflags 0x246 [ PF ZF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
```
In this example I can see the value of all registers as they were during
checkpointing and I can also see the complete command-line of my container's PID
1 process: `bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile`
在这个例子中，我可以看到所有寄存器的值，因为它们在检查点，我还可以看到容器的 PID 1 进程的完整命令行：
`bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile`。
<!--
## Summary
-->
## 总结
<!--
With the help of container checkpointing, it is possible to create a
checkpoint of a running container without stopping the container and without the
container knowing that it was checkpointed. The result of checkpointing a
container in Kubernetes is a checkpoint archive; using different tools like
`checkpointctl`, `tar`, `crit` and `gdb` the checkpoint can be analyzed. Even
with simple tools like `grep` it is possible to find information in the
checkpoint archive.
The different examples I have shown in this article how to analyze a checkpoint
are just the starting point. Depending on your requirements it is possible to
look at certain things in much more detail, but this article should give you an
introduction how to start the analysis of your checkpoint.
-->
借助容器检查点，可以在不停止容器且在容器不知情的情况下，为正在运行的容器创建检查点。
在 Kubernetes 中对容器创建一个检查点的结果是检查点存档文件；
使用不同的工具，如 `checkpointctl`、`tar`、`crit` 和 `gdb`，可以分析检查点。
即使使用像 `grep` 这样的简单工具，也可以在检查点存档中找到信息。
我在本文中展示的如何分析检查点的不同示例，这只是一个起点。
根据你的需求，可以更详细地查看某些内容，本文向你介绍了如何开始进行检查点分析。
<!--
## How do I get involved?
-->
## 如何参与？
<!--
You can reach SIG Node by several means:
-->
你可以通过多种方式联系到 SIG Node。
* Slack: [#sig-node][slack-sig-node]
* Slack: [#sig-security][slack-sig-security]
* [邮件列表][sig-node-ml]
[forensic-blog]: https://kubernetes.io/zh-cn/blog/2022/12/05/forensic-container-checkpointing-alpha/
[checkpointctl]: https://github.com/checkpoint-restore/checkpointctl
[image-files]: https://criu.org/Images
[crit]: https://criu.org/CRIT
[slack-sig-node]: https://kubernetes.slack.com/messages/sig-node
[slack-sig-security]: https://kubernetes.slack.com/messages/sig-security
[sig-node-ml]: https://groups.google.com/forum/#!forum/kubernetes-sig-node
[criu-coredump]: https://github.com/checkpoint-restore/criu/tree/criu-dev/coredump