kubernetes
/
website
mirror of https://github.com/kubernetes/website.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Clean up a policy concept page: resource-quotas

pull/48944/head
windsonsea 2024-12-04 15:45:26 +08:00
parent 7f83a3ae4c
commit 40e5543854
1 changed files with 63 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

117
content/en/docs/concepts/policy/resource-quotas.md
View File

@ -19,13 +19,14 @@ Resource quotas are a tool for administrators to address this concern.
<!-- body --> <!-- body -->
A resource quota, defined by a `ResourceQuota` object, provides constraints that limit A resource quota, defined by a `ResourceQuota` object, provides constraints that limit
aggregate resource consumption per namespace. It can limit the quantity of objects that can aggregate resource consumption per namespace. It can limit the quantity of objects that can
be created in a namespace by type, as well as the total amount of compute resources that may be created in a namespace by type, as well as the total amount of compute resources that may
be consumed by resources in that namespace. be consumed by resources in that namespace.
Resource quotas work like this: Resource quotas work like this:
- Different teams work in different namespaces. This can be enforced with [RBAC](/docs/reference/access-authn-authz/rbac/). - Different teams work in different namespaces. This can be enforced with
[RBAC](/docs/reference/access-authn-authz/rbac/).
- The administrator creates one ResourceQuota for each namespace. - The administrator creates one ResourceQuota for each namespace.
@ -36,22 +37,28 @@ Resource quotas work like this:
status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated. status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated.
- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify - If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify
requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use
the `LimitRanger` admission controller to force defaults for pods that make no compute resource requirements. the `LimitRanger` admission controller to force defaults for pods that make no compute resource requirements.
See the [walkthrough](/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/) See the [walkthrough](/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/)
for an example of how to avoid this problem. for an example of how to avoid this problem.
{{< note >}} {{< note >}}
- For `cpu` and `memory` resources, ResourceQuotas enforce that **every** - For `cpu` and `memory` resources, ResourceQuotas enforce that **every**
(new) pod in that namespace sets a limit for that resource. (new) pod in that namespace sets a limit for that resource.
If you enforce a resource quota in a namespace for either `cpu` or `memory`, If you enforce a resource quota in a namespace for either `cpu` or `memory`,
you, and other clients, **must** specify either `requests` or `limits` for that resource, you and other clients, **must** specify either `requests` or `limits` for that resource,
for every new Pod you submit. If you don't, the control plane may reject admission for every new Pod you submit. If you don't, the control plane may reject admission
for that Pod. for that Pod.
- For other resources: ResourceQuota works and will ignore pods in the namespace without setting a limit or request for that resource. It means that you can create a new pod without limit/request ephemeral storage if the resource quota limits the ephemeral storage of this namespace. - For other resources: ResourceQuota works and will ignore pods in the namespace without
setting a limit or request for that resource. It means that you can create a new pod
without limit/request for ephemeral storage if the resource quota limits the ephemeral
storage of this namespace.
You can use a [LimitRange](/docs/concepts/policy/limit-range/) to automatically set You can use a [LimitRange](/docs/concepts/policy/limit-range/) to automatically set
a default request for these resources. a default request for these resources.
{{< /note >}} {{< /note >}}
The name of a ResourceQuota object must be a valid The name of a ResourceQuota object must be a valid
@ -61,17 +68,17 @@ Examples of policies that could be created using namespaces and quotas are:
- In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores, - In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores,
let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation. let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation.
- Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace - Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace
use any amount. use any amount.
In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces,
there may be contention for resources. This is handled on a first-come-first-served basis. there may be contention for resources. This is handled on a first-come-first-served basis.
Neither contention nor changes to quota will affect already created resources. Neither contention nor changes to quota will affect already created resources.
## Enabling Resource Quota ## Enabling Resource Quota
Resource Quota support is enabled by default for many Kubernetes distributions. It is ResourceQuota support is enabled by default for many Kubernetes distributions. It is
enabled when the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} enabled when the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}}
`--enable-admission-plugins=` flag has `ResourceQuota` as `--enable-admission-plugins=` flag has `ResourceQuota` as
one of its arguments. one of its arguments.
@ -88,7 +95,7 @@ that can be requested in a given namespace.
The following resource types are supported: The following resource types are supported:
| Resource Name | Description | | Resource Name | Description |
| --------------------- | ----------------------------------------------------------- | | ------------- | ----------- |
| `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. | | `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. |
| `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. | | `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. |
| `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | | `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. |
@ -104,31 +111,31 @@ In addition to the resources mentioned above, in release 1.10, quota support for
As overcommit is not allowed for extended resources, it makes no sense to specify both `requests` As overcommit is not allowed for extended resources, it makes no sense to specify both `requests`
and `limits` for the same extended resource in a quota. So for extended resources, only quota items and `limits` for the same extended resource in a quota. So for extended resources, only quota items
with prefix `requests.` is allowed for now. with prefix `requests.` are allowed.
Take the GPU resource as an example, if the resource name is `nvidia.com/gpu`, and you want to Take the GPU resource as an example, if the resource name is `nvidia.com/gpu`, and you want to
limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows: limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows:
* `requests.nvidia.com/gpu: 4` * `requests.nvidia.com/gpu: 4`
See [Viewing and Setting Quotas](#viewing-and-setting-quotas) for more detail information. See [Viewing and Setting Quotas](#viewing-and-setting-quotas) for more details.
## Storage Resource Quota ## Storage Resource Quota
You can limit the total sum of [storage resources](/docs/concepts/storage/persistent-volumes/) that can be requested in a given namespace. You can limit the total sum of [storage resources](/docs/concepts/storage/persistent-volumes/)
that can be requested in a given namespace.
In addition, you can limit consumption of storage resources based on associated storage-class. In addition, you can limit consumption of storage resources based on associated storage-class.
| Resource Name | Description | | Resource Name | Description |
| --------------------- | ----------------------------------------------------------- | | ------------- | ----------- |
| `requests.storage` | Across all persistent volume claims, the sum of storage requests cannot exceed this value. | | `requests.storage` | Across all persistent volume claims, the sum of storage requests cannot exceed this value. |
| `persistentvolumeclaims` | The total number of [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | | `persistentvolumeclaims` | The total number of [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. |
| `<storage-class-name>.storageclass.storage.k8s.io/requests.storage` | Across all persistent volume claims associated with the `<storage-class-name>`, the sum of storage requests cannot exceed this value. | | `<storage-class-name>.storageclass.storage.k8s.io/requests.storage` | Across all persistent volume claims associated with the `<storage-class-name>`, the sum of storage requests cannot exceed this value. |
| `<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims` | Across all persistent volume claims associated with the `<storage-class-name>`, the total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | | `<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims` | Across all persistent volume claims associated with the `<storage-class-name>`, the total number of [persistent volume claims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. |
For example, if an operator wants to quota storage with `gold` storage class separate from `bronze` storage class, the operator can For example, if you want to quota storage with `gold` StorageClass separate from
define a quota as follows: a `bronze` StorageClass, you can define a quota as follows:
* `gold.storageclass.storage.k8s.io/requests.storage: 500Gi` * `gold.storageclass.storage.k8s.io/requests.storage: 500Gi`
* `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi` * `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi`
@ -136,7 +143,7 @@ define a quota as follows:
In release 1.8, quota support for local ephemeral storage is added as an alpha feature: In release 1.8, quota support for local ephemeral storage is added as an alpha feature:
| Resource Name | Description | | Resource Name | Description |
| ------------------------------- |----------------------------------------------------------- | | ------------- | ----------- |
| `requests.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value. | | `requests.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value. |
| `limits.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value. | | `limits.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value. |
| `ephemeral-storage` | Same as `requests.ephemeral-storage`. | | `ephemeral-storage` | Same as `requests.ephemeral-storage`. |
@ -169,15 +176,16 @@ Here is an example set of resources users may want to put under object count quo
* `count/cronjobs.batch` * `count/cronjobs.batch`
If you define a quota this way, it applies to Kubernetes' APIs that are part of the API server, and If you define a quota this way, it applies to Kubernetes' APIs that are part of the API server, and
to any custom resources backed by a CustomResourceDefinition. If you use [API aggregation](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) to to any custom resources backed by a CustomResourceDefinition. If you use
[API aggregation](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) to
add additional, custom APIs that are not defined as CustomResourceDefinitions, the core Kubernetes add additional, custom APIs that are not defined as CustomResourceDefinitions, the core Kubernetes
control plane does not enforce quota for the aggregated API. The extension API server is expected to control plane does not enforce quota for the aggregated API. The extension API server is expected to
provide quota enforcement if that's appropriate for the custom API. provide quota enforcement if that's appropriate for the custom API.
For example, to create a quota on a `widgets` custom resource in the `example.com` API group, use `count/widgets.example.com`. For example, to create a quota on a `widgets` custom resource in the `example.com` API group, use `count/widgets.example.com`.
When using such a resource quota (nearly for all object kinds), an object is charged When using such a resource quota (nearly for all object kinds), an object is charged
against the quota if the object kind exists (is defined) in the control plane. against the quota if the object kind exists (is defined) in the control plane.
These types of quotas are useful to protect against exhaustion of storage resources. For example, you may These types of quotas are useful to protect against exhaustion of storage resources. For example, you may
want to limit the number of Secrets in a server given their large size. Too many Secrets in a cluster can want to limit the number of Secrets in a server given their large size. Too many Secrets in a cluster can
actually prevent servers and controllers from starting. You can set a quota for Jobs to protect against actually prevent servers and controllers from starting. You can set a quota for Jobs to protect against
a poorly configured CronJob. CronJobs that create too many Jobs in a namespace can lead to a denial of service. a poorly configured CronJob. CronJobs that create too many Jobs in a namespace can lead to a denial of service.
@ -185,17 +193,17 @@ a poorly configured CronJob. CronJobs that create too many Jobs in a namespace c
There is another syntax only to set the same type of quota for certain resources. There is another syntax only to set the same type of quota for certain resources.
The following types are supported: The following types are supported:
| Resource Name | Description | | Resource Name | Description |
| ------------------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ------------- | ----------- |
| `configmaps` | The total number of ConfigMaps that can exist in the namespace. | | `configmaps` | The total number of ConfigMaps that can exist in the namespace. |
| `persistentvolumeclaims` | The total number of [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | | `persistentvolumeclaims` | The total number of [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. |
| `pods` | The total number of Pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `.status.phase in (Failed, Succeeded)` is true. | | `pods` | The total number of Pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `.status.phase in (Failed, Succeeded)` is true. |
| `replicationcontrollers` | The total number of ReplicationControllers that can exist in the namespace. | | `replicationcontrollers` | The total number of ReplicationControllers that can exist in the namespace. |
| `resourcequotas` | The total number of ResourceQuotas that can exist in the namespace. | | `resourcequotas` | The total number of ResourceQuotas that can exist in the namespace. |
| `services` | The total number of Services that can exist in the namespace. | | `services` | The total number of Services that can exist in the namespace. |
| `services.loadbalancers` | The total number of Services of type `LoadBalancer` that can exist in the namespace. | | `services.loadbalancers` | The total number of Services of type `LoadBalancer` that can exist in the namespace. |
| `services.nodeports` | The total number of `NodePorts` allocated to Services of type `NodePort` or `LoadBalancer` that can exist in the namespace. | | `services.nodeports` | The total number of `NodePorts` allocated to Services of type `NodePort` or `LoadBalancer` that can exist in the namespace. |
| `secrets` | The total number of Secrets that can exist in the namespace. | | `secrets` | The total number of Secrets that can exist in the namespace. |
For example, `pods` quota counts and enforces a maximum on the number of `pods` For example, `pods` quota counts and enforces a maximum on the number of `pods`
created in a single namespace that are not terminal. You might want to set a `pods` created in a single namespace that are not terminal. You might want to set a `pods`
@ -248,7 +256,7 @@ The `scopeSelector` supports the following values in the `operator` field:
* `DoesNotExist` * `DoesNotExist`
When using one of the following values as the `scopeName` when defining the When using one of the following values as the `scopeName` when defining the
`scopeSelector`, the `operator` must be `Exists`. `scopeSelector`, the `operator` must be `Exists`.
* `Terminating` * `Terminating`
* `NotTerminating` * `NotTerminating`
@ -470,10 +478,10 @@ have pods with affinity terms that cross namespaces. Specifically, it controls w
to set `namespaces` or `namespaceSelector` fields in pod affinity terms. to set `namespaces` or `namespaceSelector` fields in pod affinity terms.
Preventing users from using cross-namespace affinity terms might be desired since a pod Preventing users from using cross-namespace affinity terms might be desired since a pod
with anti-affinity constraints can block pods from all other namespaces with anti-affinity constraints can block pods from all other namespaces
from getting scheduled in a failure domain. from getting scheduled in a failure domain.
Using this scope operators can prevent certain namespaces (`foo-ns` in the example below) Using this scope operators can prevent certain namespaces (`foo-ns` in the example below)
from having pods that use cross-namespace pod affinity by creating a resource quota object in from having pods that use cross-namespace pod affinity by creating a resource quota object in
that namespace with `CrossNamespacePodAffinity` scope and hard limit of 0: that namespace with `CrossNamespacePodAffinity` scope and hard limit of 0:
@ -492,9 +500,9 @@ spec:
operator: Exists operator: Exists
``` ```
If operators want to disallow using `namespaces` and `namespaceSelector` by default, and If operators want to disallow using `namespaces` and `namespaceSelector` by default, and
only allow it for specific namespaces, they could configure `CrossNamespacePodAffinity` only allow it for specific namespaces, they could configure `CrossNamespacePodAffinity`
as a limited resource by setting the kube-apiserver flag --admission-control-config-file as a limited resource by setting the kube-apiserver flag `--admission-control-config-file`
to the path of the following configuration file: to the path of the following configuration file:
```yaml ```yaml
@ -513,7 +521,7 @@ plugins:
``` ```
With the above configuration, pods can use `namespaces` and `namespaceSelector` in pod affinity only With the above configuration, pods can use `namespaces` and `namespaceSelector` in pod affinity only
if the namespace where they are created have a resource quota object with if the namespace where they are created have a resource quota object with
`CrossNamespacePodAffinity` scope and a hard limit greater than or equal to the number of pods using those fields. `CrossNamespacePodAffinity` scope and a hard limit greater than or equal to the number of pods using those fields.
## Requests compared to Limits {#requests-vs-limits} ## Requests compared to Limits {#requests-vs-limits}
@ -522,12 +530,12 @@ When allocating compute resources, each container may specify a request and a li
The quota can be configured to quota either value. The quota can be configured to quota either value.
If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming
container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`, container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`,
then it requires that every incoming container specifies an explicit limit for those resources. then it requires that every incoming container specifies an explicit limit for those resources.
## Viewing and Setting Quotas ## Viewing and Setting Quotas
Kubectl supports creating, updating, and viewing quotas: kubectl supports creating, updating, and viewing quotas:
```shell ```shell
kubectl create namespace myspace kubectl create namespace myspace
@ -619,7 +627,7 @@ services 0 10
services.loadbalancers 0 2 services.loadbalancers 0 2
``` ```
Kubectl also supports object count quota for all standard namespaced resources kubectl also supports object count quota for all standard namespaced resources
using the syntax `count/<resource>.<group>`: using the syntax `count/<resource>.<group>`:
```shell ```shell
@ -652,7 +660,7 @@ count/secrets 1 4
## Quota and Cluster Capacity ## Quota and Cluster Capacity
ResourceQuotas are independent of the cluster capacity. They are ResourceQuotas are independent of the cluster capacity. They are
expressed in absolute units. So, if you add nodes to your cluster, this does *not* expressed in absolute units. So, if you add nodes to your cluster, this does *not*
automatically give each namespace the ability to consume more resources. automatically give each namespace the ability to consume more resources.
Sometimes more complex policies may be desired, such as: Sometimes more complex policies may be desired, such as:
@ -671,7 +679,7 @@ restrictions around nodes: pods from several namespaces may run on the same node
## Limit Priority Class consumption by default ## Limit Priority Class consumption by default
It may be desired that pods at a particular priority, eg. "cluster-services", It may be desired that pods at a particular priority, such as "cluster-services",
should be allowed in a namespace, if and only if, a matching quota object exists. should be allowed in a namespace, if and only if, a matching quota object exists.
With this mechanism, operators are able to restrict usage of certain high With this mechanism, operators are able to restrict usage of certain high
@ -711,9 +719,9 @@ resourcequota/pods-cluster-services created
In this case, a pod creation will be allowed if: In this case, a pod creation will be allowed if:
1. the Pod's `priorityClassName` is not specified. 1. the Pod's `priorityClassName` is not specified.
1. the Pod's `priorityClassName` is specified to a value other than `cluster-services`. 1. the Pod's `priorityClassName` is specified to a value other than `cluster-services`.
1. the Pod's `priorityClassName` is set to `cluster-services`, it is to be created 1. the Pod's `priorityClassName` is set to `cluster-services`, it is to be created
in the `kube-system` namespace, and it has passed the resource quota check. in the `kube-system` namespace, and it has passed the resource quota check.
A Pod creation request is rejected if its `priorityClassName` is set to `cluster-services` A Pod creation request is rejected if its `priorityClassName` is set to `cluster-services`
@ -721,7 +729,8 @@ and it is to be created in a namespace other than `kube-system`.
## {{% heading "whatsnext" %}} ## {{% heading "whatsnext" %}}
- See [ResourceQuota design doc](https://git.k8s.io/design-proposals-archive/resource-management/admission_control_resource_quota.md) for more information. - See [ResourceQuota design document](https://git.k8s.io/design-proposals-archive/resource-management/admission_control_resource_quota.md)
for more information.
- See a [detailed example for how to use resource quota](/docs/tasks/administer-cluster/quota-api-object/). - See a [detailed example for how to use resource quota](/docs/tasks/administer-cluster/quota-api-object/).
- Read [Quota support for priority class design doc](https://git.k8s.io/design-proposals-archive/scheduling/pod-priority-resourcequota.md). - Read [Quota support for priority class design document](https://git.k8s.io/design-proposals-archive/scheduling/pod-priority-resourcequota.md).
- See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765) - See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765).