From 2c45e1ae2885e53e80b85e5ea3a6d39b9e33e10c Mon Sep 17 00:00:00 2001 From: Shu Muto Date: Sat, 13 Jul 2024 08:36:55 +0900 Subject: [PATCH] Add caution for using memory-backed emptydir (#44949) * Add caution for using memory-backed emptydir To use default size of memory-backed emptydir, some risks exist. So this adds cautions for the risks. * Fix sub section title Co-authored-by: Tim Bannister * Move conciderations for memory backed emptyDir into Resource Management for Pods and Container page. * Revirt removing spaces on volumes.md * Revirt removing spaces on manage-resources-containers.md * Fixed to make it more readable and concise --------- Co-authored-by: Tim Bannister --- .../manage-resources-containers.md | 47 ++++++++++++++++++- content/en/docs/concepts/storage/volumes.md | 12 +++-- 2 files changed, 55 insertions(+), 4 deletions(-) diff --git a/content/en/docs/concepts/configuration/manage-resources-containers.md b/content/en/docs/concepts/configuration/manage-resources-containers.md index 4d21ec9950..99dc662aa1 100644 --- a/content/en/docs/concepts/configuration/manage-resources-containers.md +++ b/content/en/docs/concepts/configuration/manage-resources-containers.md @@ -215,7 +215,8 @@ limits you defined. as restartable, Kubernetes restarts the container. - The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an `emptyDir`. The kubelet tracks `tmpfs` emptyDir volumes as container - memory use, rather than as local ephemeral storage. + memory use, rather than as local ephemeral storage.　When using memory backed `emptyDir`, + be sure to check the notes [below](#memory-backed-emptydir). If a container exceeds its memory request and the node that it runs on becomes short of memory overall, it is likely that the Pod the container belongs to will be @@ -237,6 +238,50 @@ are available in your cluster, then Pod resource usage can be retrieved either from the [Metrics API](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api) directly or from your monitoring tools. +### Considerations for memory backed `emptyDir` volumes {#memory-backed-emptydir} + +{{< caution >}} +If you do not specify a `sizeLimit` for an `emptyDir` volume, that volume may +consume up to that pod's memory limit (`Pod.spec.containers[].resources.limits.memory`). +If you do not set a memory limit, the pod has no upper bound on memory consumption, +and can consume all available memory on the node. Kubernetes schedules pods based +on resource requests (`Pod.spec.containers[].resources.requests`) and will not +consider memory usage above the request when deciding if another pod can fit on +a given node. This can result in a denial of service and cause the OS to do +out-of-memory (OOM) handling. It is possible to create any number of `emptyDir`s +that could potentially consume all available memory on the node, making OOM +more likely. +{{< /caution >}} + +From the perspective of memory management, there are some similarities between +when a process uses memory as a work area and when using memory-backed +`emptyDir`. But when using memory as a volume like memory-backed `emptyDir`, +there are additional points below that you should be careful of. + +* Files stored on a memory-backed volume are almost entirely managed by the + user application. Unlike when used as a work area for a process, you can not + rely on things like language-level garbage collection. +* The purpose of writing files to a volume is to save data or pass it between + applications. Neither Kubernetes nor the OS may automatically delete files + from a volume, so memory used by those files can not be reclaimed when the + system or the pod are under memory pressure. +* A memory-backed `emptyDir` is useful because of its performance, but memory + is generally much smaller in size and much higher in cost than other storage + media, such as disks or SSDs. Using large amounts of memory for `emptyDir` + volumes may affect the normal operation of your pod or of the whole node, + so should be used carefully. + +If you are administering a cluster or namespace, you can also set +[ResourceQuota](/docs/concepts/policy/resource-quotas/) that limits memory use; +you may also want to define a [LimitRange](/docs/concepts/policy/limit-range/) +for additional enforcement. +If you specify a `spec.containers[].resources.limits.memory` for each Pod, +then the muximum size of an `emptyDir` volume will be the pod's memory limit. + +As an alternative, a cluster administrator can enforce size limits for +`emptyDir` volumes in new Pods using a policy mechanism such as +[ValidationAdmissionPolicy](/docs/reference/access-authn-authz/validating-admission-policy). + ## Local ephemeral storage diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index 3d9c4b3bfd..bb627d4918 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -262,11 +262,17 @@ If that is filled up from another source (for example, log files or image overlays), the `emptyDir` may run out of capacity before this limit. {{< note >}} -If the `SizeMemoryBackedVolumes` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled, -you can specify a size for memory backed volumes. If no size is specified, memory -backed volumes are sized to node allocatable memory. +You can specify a size for memory backed volumes, provided that the `SizeMemoryBackedVolumes` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +is enabled in your cluster (this has been beta, and active by default, since the Kubernetes 1.22 release). +If you don't specify a volume size, memory backed volumes are sized to node allocatable memory. {{< /note>}} +{{< caution >}} +Please check [here](/docs/concepts/configuration/manage-resources-containers/#memory-backed-emptydir) +for points to note in terms of resource management when using memory-backed `emptyDir`. +{{< /caution >}} + #### emptyDir configuration example ```yaml