diff --git a/content/ja/docs/tutorials/clusters/_index.md b/content/ja/docs/tutorials/clusters/_index.md new file mode 100755 index 0000000000..2db6156b6f --- /dev/null +++ b/content/ja/docs/tutorials/clusters/_index.md @@ -0,0 +1,5 @@ +--- +title: "Clusters" +weight: 60 +--- + diff --git a/content/ja/docs/tutorials/clusters/apparmor.md b/content/ja/docs/tutorials/clusters/apparmor.md new file mode 100644 index 0000000000..54c8a0f44c --- /dev/null +++ b/content/ja/docs/tutorials/clusters/apparmor.md @@ -0,0 +1,473 @@ +--- +reviewers: +- stclair +title: Restrict a Container's Access to Resources with AppArmor +content_type: tutorial +weight: 10 +--- + + + +{{< feature-state for_k8s_version="v1.4" state="beta" >}} + + +AppArmor is a Linux kernel security module that supplements the standard Linux user and group based +permissions to confine programs to a limited set of resources. AppArmor can be configured for any +application to reduce its potential attack surface and provide greater in-depth defense. It is +configured through profiles tuned to allow the access needed by a specific program or container, +such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either +*enforcing* mode, which blocks access to disallowed resources, or *complain* mode, which only reports +violations. + +AppArmor can help you to run a more secure deployment by restricting what containers are allowed to +do, and/or provide better auditing through system logs. However, it is important to keep in mind +that AppArmor is not a silver bullet and can only do so much to protect against exploits in your +application code. It is important to provide good, restrictive profiles, and harden your +applications and cluster from other angles as well. + + + +## {{% heading "objectives" %}} + + +* See an example of how to load a profile on a node +* Learn how to enforce the profile on a Pod +* Learn how to check that the profile is loaded +* See what happens when a profile is violated +* See what happens when a profile cannot be loaded + + + +## {{% heading "prerequisites" %}} + + +Make sure: + +1. Kubernetes version is at least v1.4 -- Kubernetes support for AppArmor was added in + v1.4. Kubernetes components older than v1.4 are not aware of the new AppArmor annotations, and + will **silently ignore** any AppArmor settings that are provided. To ensure that your Pods are + receiving the expected protections, it is important to verify the Kubelet version of your nodes: + + ```shell + kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.kubeletVersion}\n{end}' + ``` + ``` + gke-test-default-pool-239f5d02-gyn2: v1.4.0 + gke-test-default-pool-239f5d02-x1kf: v1.4.0 + gke-test-default-pool-239f5d02-xwux: v1.4.0 + ``` + +2. AppArmor kernel module is enabled -- For the Linux kernel to enforce an AppArmor profile, the + AppArmor kernel module must be installed and enabled. Several distributions enable the module by + default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the + module is enabled, check the `/sys/module/apparmor/parameters/enabled` file: + + ```shell + cat /sys/module/apparmor/parameters/enabled + Y + ``` + + If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor + options if the kernel module is not enabled. + + {{< note >}} + Ubuntu carries many AppArmor patches that have not been merged into the upstream Linux + kernel, including patches that add additional hooks and features. Kubernetes has only been + tested with the upstream version, and does not promise support for other features. + {{< /note >}} + +3. Container runtime supports AppArmor -- Currently all common Kubernetes-supported container + runtimes should support AppArmor, like {{< glossary_tooltip term_id="docker">}}, + {{< glossary_tooltip term_id="cri-o" >}} or {{< glossary_tooltip term_id="containerd" >}}. + Please refer to the corresponding runtime documentation and verify that the cluster fulfills + the requirements to use AppArmor. + +4. Profile is loaded -- AppArmor is applied to a Pod by specifying an AppArmor profile that each + container should be run with. If any of the specified profiles is not already loaded in the + kernel, the Kubelet (>= v1.4) will reject the Pod. You can view which profiles are loaded on a + node by checking the `/sys/kernel/security/apparmor/profiles` file. For example: + + ```shell + ssh gke-test-default-pool-239f5d02-gyn2 "sudo cat /sys/kernel/security/apparmor/profiles | sort" + ``` + ``` + apparmor-test-deny-write (enforce) + apparmor-test-audit-write (enforce) + docker-default (enforce) + k8s-nginx (enforce) + ``` + + For more details on loading profiles on nodes, see + [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + +As long as the Kubelet version includes AppArmor support (>= v1.4), the Kubelet will reject a Pod +with AppArmor options if any of the prerequisites are not met. You can also verify AppArmor support +on nodes by checking the node ready condition message (though this is likely to be removed in a +later release): + +```shell +kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}\n{end}' +``` +``` +gke-test-default-pool-239f5d02-gyn2: kubelet is posting ready status. AppArmor enabled +gke-test-default-pool-239f5d02-x1kf: kubelet is posting ready status. AppArmor enabled +gke-test-default-pool-239f5d02-xwux: kubelet is posting ready status. AppArmor enabled +``` + + + + + +## Securing a Pod + +{{< note >}} +AppArmor is currently in beta, so options are specified as annotations. Once support graduates to +general availability, the annotations will be replaced with first-class fields (more details in +[Upgrade path to GA](#upgrade-path-to-general-availability)). +{{< /note >}} + +AppArmor profiles are specified *per-container*. To specify the AppArmor profile to run a Pod +container with, add an annotation to the Pod's metadata: + +```yaml +container.apparmor.security.beta.kubernetes.io/: +``` + +Where `` is the name of the container to apply the profile to, and `` +specifies the profile to apply. The `profile_ref` can be one of: + +* `runtime/default` to apply the runtime's default profile +* `localhost/` to apply the profile loaded on the host with the name `` +* `unconfined` to indicate that no profiles will be loaded + +See the [API Reference](#api-reference) for the full details on the annotation and profile name formats. + +Kubernetes AppArmor enforcement works by first checking that all the prerequisites have been +met, and then forwarding the profile selection to the container runtime for enforcement. If the +prerequisites have not been met, the Pod will be rejected, and will not run. + +To verify that the profile was applied, you can look for the AppArmor security option listed in the container created event: + +```shell +kubectl get events | grep Created +``` +``` +22s 22s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet e2e-test-stclair-node-pool-31nt} Created container with docker id 269a53b202d3; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] +``` + +You can also verify directly that the container's root process is running with the correct profile by checking its proc attr: + +```shell +kubectl exec cat /proc/1/attr/current +``` +``` +k8s-apparmor-example-deny-write (enforce) +``` + +## Example + +*This example assumes you have already set up a cluster with AppArmor support.* + +First, we need to load the profile we want to use onto our nodes. This profile denies all file writes: + +```shell +#include + +profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { + #include + + file, + + # Deny all file writes. + deny /** w, +} +``` + +Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our +nodes. For this example we'll just use SSH to install the profiles, but other approaches are +discussed in [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + +```shell +NODES=( + # The SSH-accessible domain names of your nodes + gke-test-default-pool-239f5d02-gyn2.us-central1-a.my-k8s + gke-test-default-pool-239f5d02-x1kf.us-central1-a.my-k8s + gke-test-default-pool-239f5d02-xwux.us-central1-a.my-k8s) +for NODE in ${NODES[*]}; do ssh $NODE 'sudo apparmor_parser -q < + +profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { + #include + + file, + + # Deny all file writes. + deny /** w, +} +EOF' +done +``` + +Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: + +{{< codenew file="pods/security/hello-apparmor.yaml" >}} + +```shell +kubectl create -f ./hello-apparmor.yaml +``` + +If we look at the pod events, we can see that the Pod container was created with the AppArmor +profile "k8s-apparmor-example-deny-write": + +```shell +kubectl get events | grep hello-apparmor +``` +``` +14s 14s 1 hello-apparmor Pod Normal Scheduled {default-scheduler } Successfully assigned hello-apparmor to gke-test-default-pool-239f5d02-gyn2 +14s 14s 1 hello-apparmor Pod spec.containers{hello} Normal Pulling {kubelet gke-test-default-pool-239f5d02-gyn2} pulling image "busybox" +13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Pulled {kubelet gke-test-default-pool-239f5d02-gyn2} Successfully pulled image "busybox" +13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet gke-test-default-pool-239f5d02-gyn2} Created container with docker id 06b6cd1c0989; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] +13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Started {kubelet gke-test-default-pool-239f5d02-gyn2} Started container with docker id 06b6cd1c0989 +``` + +We can verify that the container is actually running with that profile by checking its proc attr: + +```shell +kubectl exec hello-apparmor cat /proc/1/attr/current +``` +``` +k8s-apparmor-example-deny-write (enforce) +``` + +Finally, we can see what happens if we try to violate the profile by writing to a file: + +```shell +kubectl exec hello-apparmor touch /tmp/test +``` +``` +touch: /tmp/test: Permission denied +error: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1 +``` + +To wrap up, let's look at what happens if we try to specify a profile that hasn't been loaded: + +```shell +kubectl create -f /dev/stdin < +Annotations: container.apparmor.security.beta.kubernetes.io/hello=localhost/k8s-apparmor-example-allow-write +Status: Pending +Reason: AppArmor +Message: Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded +IP: +Controllers: +Containers: + hello: + Container ID: + Image: busybox + Image ID: + Port: + Command: + sh + -c + echo 'Hello AppArmor!' && sleep 1h + State: Waiting + Reason: Blocked + Ready: False + Restart Count: 0 + Environment: + Mounts: + /var/run/secrets/kubernetes.io/serviceaccount from default-token-dnz7v (ro) +Conditions: + Type Status + Initialized True + Ready False + PodScheduled True +Volumes: + default-token-dnz7v: + Type: Secret (a volume populated by a Secret) + SecretName: default-token-dnz7v + Optional: false +QoS Class: BestEffort +Node-Selectors: +Tolerations: +Events: + FirstSeen LastSeen Count From SubobjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 23s 23s 1 {default-scheduler } Normal Scheduled Successfully assigned hello-apparmor-2 to e2e-test-stclair-node-pool-t1f5 + 23s 23s 1 {kubelet e2e-test-stclair-node-pool-t1f5} Warning AppArmor Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded +``` + +Note the pod status is Failed, with a helpful error message: `Pod Cannot enforce AppArmor: profile +"k8s-apparmor-example-allow-write" is not loaded`. An event was also recorded with the same message. + +## Administration + +### Setting up nodes with profiles + +Kubernetes does not currently provide any native mechanisms for loading AppArmor profiles onto +nodes. There are lots of ways to setup the profiles though, such as: + +* Through a [DaemonSet](/docs/concepts/workloads/controllers/daemonset/) that runs a Pod on each node to + ensure the correct profiles are loaded. An example implementation can be found + [here](https://git.k8s.io/kubernetes/test/images/apparmor-loader). +* At node initialization time, using your node initialization scripts (e.g. Salt, Ansible, etc.) or + image. +* By copying the profiles to each node and loading them through SSH, as demonstrated in the + [Example](#example). + +The scheduler is not aware of which profiles are loaded onto which node, so the full set of profiles +must be loaded onto every node. An alternative approach is to add a node label for each profile (or +class of profiles) on the node, and use a +[node selector](/docs/concepts/scheduling-eviction/assign-pod-node/) to ensure the Pod is run on a +node with the required profile. + +### Restricting profiles with the PodSecurityPolicy + +If the PodSecurityPolicy extension is enabled, cluster-wide AppArmor restrictions can be applied. To +enable the PodSecurityPolicy, the following flag must be set on the `apiserver`: + +``` +--enable-admission-plugins=PodSecurityPolicy[,others...] +``` + +The AppArmor options can be specified as annotations on the PodSecurityPolicy: + +```yaml +apparmor.security.beta.kubernetes.io/defaultProfileName: +apparmor.security.beta.kubernetes.io/allowedProfileNames: [,others...] +``` + +The default profile name option specifies the profile to apply to containers by default when none is +specified. The allowed profile names option specifies a list of profiles that Pod containers are +allowed to be run with. If both options are provided, the default must be allowed. The profiles are +specified in the same format as on containers. See the [API Reference](#api-reference) for the full +specification. + +### Disabling AppArmor + +If you do not want AppArmor to be available on your cluster, it can be disabled by a command-line flag: + +``` +--feature-gates=AppArmor=false +``` + +When disabled, any Pod that includes an AppArmor profile will fail validation with a "Forbidden" +error. Note that by default docker always enables the "docker-default" profile on non-privileged +pods (if the AppArmor kernel module is enabled), and will continue to do so even if the feature-gate +is disabled. The option to disable AppArmor will be removed when AppArmor graduates to general +availability (GA). + +### Upgrading to Kubernetes v1.4 with AppArmor + +No action is required with respect to AppArmor to upgrade your cluster to v1.4. However, if any +existing pods had an AppArmor annotation, they will not go through validation (or PodSecurityPolicy +admission). If permissive profiles are loaded on the nodes, a malicious user could pre-apply a +permissive profile to escalate the pod privileges above the docker-default. If this is a concern, it +is recommended to scrub the cluster of any pods containing an annotation with +`apparmor.security.beta.kubernetes.io`. + +### Upgrade path to General Availability + +When AppArmor is ready to be graduated to general availability (GA), the options currently specified +through annotations will be converted to fields. Supporting all the upgrade and downgrade paths +through the transition is very nuanced, and will be explained in detail when the transition +occurs. We will commit to supporting both fields and annotations for at least 2 releases, and will +explicitly reject the annotations for at least 2 releases after that. + +## Authoring Profiles + +Getting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some +tools to help with that: + +* `aa-genprof` and `aa-logprof` generate profile rules by monitoring an application's activity and + logs, and admitting the actions it takes. Further instructions are provided by the + [AppArmor documentation](https://gitlab.com/apparmor/apparmor/wikis/Profiling_with_tools). +* [bane](https://github.com/jfrazelle/bane) is an AppArmor profile generator for Docker that uses a + simplified profile language. + +It is recommended to run your application through Docker on a development workstation to generate +the profiles, but there is nothing preventing running the tools on the Kubernetes node where your +Pod is running. + +To debug problems with AppArmor, you can check the system logs to see what, specifically, was +denied. AppArmor logs verbose messages to `dmesg`, and errors can usually be found in the system +logs or through `journalctl`. More information is provided in +[AppArmor failures](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Failures). + + +## API Reference + +### Pod Annotation + +Specifying the profile a container will run with: + +- **key**: `container.apparmor.security.beta.kubernetes.io/` + Where `` matches the name of a container in the Pod. + A separate profile can be specified for each container in the Pod. +- **value**: a profile reference, described below + +### Profile Reference + +- `runtime/default`: Refers to the default runtime profile. + - Equivalent to not specifying a profile (without a PodSecurityPolicy default), except it still + requires AppArmor to be enabled. + - For Docker, this resolves to the + [`docker-default`](https://docs.docker.com/engine/security/apparmor/) profile for non-privileged + containers, and unconfined (no profile) for privileged containers. +- `localhost/`: Refers to a profile loaded on the node (localhost) by name. + - The possible profile names are detailed in the + [core policy reference](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Core_Policy_Reference#profile-names-and-attachment-specifications). +- `unconfined`: This effectively disables AppArmor on the container. + +Any other profile reference format is invalid. + +### PodSecurityPolicy Annotations + +Specifying the default profile to apply to containers when none is provided: + +* **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` +* **value**: a profile reference, described above + +Specifying the list of profiles Pod containers is allowed to specify: + +* **key**: `apparmor.security.beta.kubernetes.io/allowedProfileNames` +* **value**: a comma-separated list of profile references (described above) + - Although an escaped comma is a legal character in a profile name, it cannot be explicitly + allowed here. + + + +## {{% heading "whatsnext" %}} + + +Additional resources: + +* [Quick guide to the AppArmor profile language](https://gitlab.com/apparmor/apparmor/wikis/QuickProfileLanguage) +* [AppArmor core policy reference](https://gitlab.com/apparmor/apparmor/wikis/Policy_Layout) + +