Merge pull request #641 from derekwaynecarr/quota_13
Improve and update ResourceQuota for 1.3pull/766/head
commit
eb90fcf886
|
@ -0,0 +1,9 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ResourceQuota
|
||||||
|
metadata:
|
||||||
|
name: best-effort
|
||||||
|
spec:
|
||||||
|
hard:
|
||||||
|
pods: "10"
|
||||||
|
scopes:
|
||||||
|
- BestEffort
|
|
@ -0,0 +1,11 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ResourceQuota
|
||||||
|
metadata:
|
||||||
|
name: compute-resources
|
||||||
|
spec:
|
||||||
|
hard:
|
||||||
|
pods: "4"
|
||||||
|
requests.cpu: "1"
|
||||||
|
requests.memory: 1Gi
|
||||||
|
limits.cpu: "2"
|
||||||
|
limits.memory: 2Gi
|
|
@ -4,29 +4,26 @@
|
||||||
When several users or teams share a cluster with a fixed number of nodes,
|
When several users or teams share a cluster with a fixed number of nodes,
|
||||||
there is a concern that one team could use more than its fair share of resources.
|
there is a concern that one team could use more than its fair share of resources.
|
||||||
|
|
||||||
Resource quotas are a tool for administrators to address this concern. Resource quotas
|
Resource quotas are a tool for administrators to address this concern.
|
||||||
work like this:
|
|
||||||
|
A resource quota, defined by a `ResourceQuota` object, provides constraints that limit
|
||||||
|
aggregate resource consumption per namespace. It can limit the quantity of objects that can
|
||||||
|
be created in a namespace by type, as well as the total amount of compute resources that may
|
||||||
|
be consumed by resources in that project.
|
||||||
|
|
||||||
|
Resource quotas work like this:
|
||||||
|
|
||||||
- Different teams work in different namespaces. Currently this is voluntary, but
|
- Different teams work in different namespaces. Currently this is voluntary, but
|
||||||
support for making this mandatory via ACLs is planned.
|
support for making this mandatory via ACLs is planned.
|
||||||
- The administrator creates a Resource Quota for each namespace.
|
- The administrator creates one or more Resource Quota objects for each namespace.
|
||||||
- Users put compute resource requests on their pods. The sum of all resource requests across
|
- Users create resources (pods, services, etc.) in the namespace, and the quota system
|
||||||
all pods in the same namespace must not exceed any hard resource limit in any Resource Quota
|
tracks usage to ensure it does not exceed hard resource limits defined in a Resource Quota.
|
||||||
document for the namespace. Note that we used to verify Resource Quota by taking the sum of
|
- If creating or updating a resource violates a quota constraint, the request will fail with HTTP
|
||||||
resource limits of the pods, but this was altered to use resource requests. Backwards compatibility
|
status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated.
|
||||||
for those pods previously created is preserved because pods that only specify a resource limit have
|
- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify
|
||||||
their resource requests defaulted to match their defined limits. The user is only charged for the
|
requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use
|
||||||
resources they request in the Resource Quota versus their limits because the request is the minimum
|
the LimitRange admission controller to force defaults for pods that make no compute resource requirements.
|
||||||
amount of resource guaranteed by the cluster during scheduling. For more information on over commit,
|
See the [walkthrough](/docs/admin/resourcequota/walkthrough/) for an example to avoid this problem.
|
||||||
see [compute-resources](/docs/user-guide/compute-resources).
|
|
||||||
- If creating a pod would cause the namespace to exceed any of the limits specified in the
|
|
||||||
the Resource Quota for that namespace, then the request will fail with HTTP status
|
|
||||||
code `403 FORBIDDEN`.
|
|
||||||
- If quota is enabled in a namespace and the user does not specify *requests* on the pod for each
|
|
||||||
of the resources for which quota is enabled, then the POST of the pod will fail with HTTP
|
|
||||||
status code `403 FORBIDDEN`. Hint: Use the LimitRange admission controller to force default
|
|
||||||
values of *limits* (then resource *requests* would be equal to *limits* by default, see
|
|
||||||
[admission controller](/docs/admin/admission-controllers)) before the quota is checked to avoid this problem.
|
|
||||||
|
|
||||||
Examples of policies that could be created using namespaces and quotas are:
|
Examples of policies that could be created using namespaces and quotas are:
|
||||||
|
|
||||||
|
@ -38,7 +35,7 @@ Examples of policies that could be created using namespaces and quotas are:
|
||||||
In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces,
|
In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces,
|
||||||
there may be contention for resources. This is handled on a first-come-first-served basis.
|
there may be contention for resources. This is handled on a first-come-first-served basis.
|
||||||
|
|
||||||
Neither contention nor changes to quota will affect already-running pods.
|
Neither contention nor changes to quota will affect already created resources.
|
||||||
|
|
||||||
## Enabling Resource Quota
|
## Enabling Resource Quota
|
||||||
|
|
||||||
|
@ -57,11 +54,12 @@ in a namespace can be limited. The following compute resource types are support
|
||||||
|
|
||||||
| Resource Name | Description |
|
| Resource Name | Description |
|
||||||
| ------------ | ----------- |
|
| ------------ | ----------- |
|
||||||
| cpu | Total cpu requests of containers |
|
| `cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. |
|
||||||
| memory | Total memory requests of containers
|
| `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. |
|
||||||
|
| `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. |
|
||||||
For example, `cpu` quota sums up the `resources.requests.cpu` fields of every
|
| `memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. |
|
||||||
container of every pod in the namespace, and enforces a maximum on that sum.
|
| `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. |
|
||||||
|
| `requests.memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. |
|
||||||
|
|
||||||
## Object Count Quota
|
## Object Count Quota
|
||||||
|
|
||||||
|
@ -70,12 +68,15 @@ are supported:
|
||||||
|
|
||||||
| Resource Name | Description |
|
| Resource Name | Description |
|
||||||
| ------------ | ----------- |
|
| ------------ | ----------- |
|
||||||
| pods | Total number of pods |
|
| `configmaps` | The total number of config maps that can exist in the namespace. |
|
||||||
| services | Total number of services |
|
| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. |
|
||||||
| replicationcontrollers | Total number of replication controllers |
|
| `pods` | The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `status.phase in (Failed, Succeeded)` is true. |
|
||||||
| resourcequotas | Total number of [resource quotas](/docs/admin/admission-controllers/#resourcequota) |
|
| `replicationcontrollers` | The total number of replication controllers that can exist in the namespace. |
|
||||||
| secrets | Total number of secrets |
|
| `resourcequotas` | The total number of [resource quotas](/docs/admin/admission-controllers/#resourcequota) that can exist in the namespace. |
|
||||||
| persistentvolumeclaims | Total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) |
|
| `services` | The total number of services that can exist in the namespace. |
|
||||||
|
| `services.loadbalancers` | The total number of services of type load balancer that can exist in the namespace. |
|
||||||
|
| `services.nodeports` | The total number of services of type node port that can exist in the namespace. |
|
||||||
|
| `secrets` | The total number of secrets that can exist in the namespace. |
|
||||||
|
|
||||||
For example, `pods` quota counts and enforces a maximum on the number of `pods`
|
For example, `pods` quota counts and enforces a maximum on the number of `pods`
|
||||||
created in a single namespace.
|
created in a single namespace.
|
||||||
|
@ -84,45 +85,107 @@ You might want to set a pods quota on a namespace
|
||||||
to avoid the case where a user creates many small pods and exhausts the cluster's
|
to avoid the case where a user creates many small pods and exhausts the cluster's
|
||||||
supply of Pod IPs.
|
supply of Pod IPs.
|
||||||
|
|
||||||
|
## Quota Scopes
|
||||||
|
|
||||||
|
Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches
|
||||||
|
the intersection of enumerated scopes.
|
||||||
|
|
||||||
|
When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope.
|
||||||
|
Resources specified on the quota outside of the allowed set results in a validation error.
|
||||||
|
|
||||||
|
| Scope | Description |
|
||||||
|
| ----- | ----------- |
|
||||||
|
| `Terminating` | Match pods where `spec.activeDeadlineSeconds >= 0` |
|
||||||
|
| `NotTerminating` | Match pods where `spec.activeDeadlineSeconds is nil` |
|
||||||
|
| `BestEffort` | Match pods that have best effort quality of service. |
|
||||||
|
| `NotBestEffort` | Match pods that do not have best effort quality of service. |
|
||||||
|
|
||||||
|
The `BestEffort` scope restricts a quota to tracking the following resource: `pods`
|
||||||
|
|
||||||
|
The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota to tracking the following resources:
|
||||||
|
|
||||||
|
* `cpu`
|
||||||
|
* `limits.cpu`
|
||||||
|
* `limits.memory`
|
||||||
|
* `memory`
|
||||||
|
* `pods`
|
||||||
|
* `requests.cpu`
|
||||||
|
* `requests.memory`
|
||||||
|
|
||||||
|
## Requests vs Limits
|
||||||
|
|
||||||
|
When allocating compute resources, each container may specify a request and a limit value for either CPU or memory.
|
||||||
|
The quota can be configured to quota either value.
|
||||||
|
|
||||||
|
If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming
|
||||||
|
container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`,
|
||||||
|
then it requires that every incoming container specifies an explict limit for those resources.
|
||||||
|
|
||||||
## Viewing and Setting Quotas
|
## Viewing and Setting Quotas
|
||||||
|
|
||||||
Kubectl supports creating, updating, and viewing quotas:
|
Kubectl supports creating, updating, and viewing quotas:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl namespace myspace
|
$ kubectl create namespace myspace
|
||||||
$ cat <<EOF > quota.json
|
|
||||||
{
|
$ cat <<EOF > compute-resources.yaml
|
||||||
"apiVersion": "v1",
|
apiVersion: v1
|
||||||
"kind": "ResourceQuota",
|
kind: ResourceQuota
|
||||||
"metadata": {
|
metadata:
|
||||||
"name": "quota"
|
name: compute-resources
|
||||||
},
|
spec:
|
||||||
"spec": {
|
hard:
|
||||||
"hard": {
|
pods: "4"
|
||||||
"memory": "1Gi",
|
requests.cpu: "1"
|
||||||
"cpu": "20",
|
requests.memory: 1Gi
|
||||||
"pods": "10",
|
limits.cpu: "2"
|
||||||
"services": "5",
|
limits.memory: 2Gi
|
||||||
"replicationcontrollers":"20",
|
|
||||||
"resourcequotas":"1"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
EOF
|
EOF
|
||||||
$ kubectl create -f ./quota.json
|
$ kubectl create -f ./compute-resources.yaml --namespace=myspace
|
||||||
$ kubectl get quota
|
|
||||||
NAME
|
$ cat <<EOF > object-counts.yaml
|
||||||
quota
|
apiVersion: v1
|
||||||
$ kubectl describe quota quota
|
kind: ResourceQuota
|
||||||
Name: quota
|
metadata:
|
||||||
|
name: object-counts
|
||||||
|
spec:
|
||||||
|
hard:
|
||||||
|
configmaps: "10"
|
||||||
|
persistentvolumeclaims: "4"
|
||||||
|
replicationcontrollers: "20"
|
||||||
|
secrets: "10"
|
||||||
|
services: "10"
|
||||||
|
services.loadbalancers: "2"
|
||||||
|
EOF
|
||||||
|
$ kubectl create -f ./object-counts.yaml --namespace=myspace
|
||||||
|
|
||||||
|
$ kubectl get quota --namespace=myspace
|
||||||
|
NAME AGE
|
||||||
|
compute-resources 30s
|
||||||
|
object-counts 32s
|
||||||
|
|
||||||
|
$ kubectl describe quota compute-resources --namespace=myspace
|
||||||
|
Name: compute-resources
|
||||||
|
Namespace: myspace
|
||||||
Resource Used Hard
|
Resource Used Hard
|
||||||
-------- ---- ----
|
-------- ---- ----
|
||||||
cpu 0m 20
|
limits.cpu 0 2
|
||||||
memory 0 1Gi
|
limits.memory 0 2Gi
|
||||||
pods 5 10
|
pods 0 4
|
||||||
replicationcontrollers 5 20
|
requests.cpu 0 1
|
||||||
resourcequotas 1 1
|
requests.memory 0 1Gi
|
||||||
services 3 5
|
|
||||||
|
$ kubectl describe quota object-counts --namespace=myspace
|
||||||
|
Name: object-counts
|
||||||
|
Namespace: myspace
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
configmaps 0 10
|
||||||
|
persistentvolumeclaims 0 4
|
||||||
|
replicationcontrollers 0 20
|
||||||
|
secrets 1 10
|
||||||
|
services 0 10
|
||||||
|
services.loadbalancers 0 2
|
||||||
```
|
```
|
||||||
|
|
||||||
## Quota and Cluster Capacity
|
## Quota and Cluster Capacity
|
||||||
|
|
|
@ -0,0 +1,13 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ResourceQuota
|
||||||
|
metadata:
|
||||||
|
name: not-best-effort
|
||||||
|
spec:
|
||||||
|
hard:
|
||||||
|
pods: "4"
|
||||||
|
requests.cpu: "1"
|
||||||
|
requests.memory: 1Gi
|
||||||
|
limits.cpu: "2"
|
||||||
|
limits.memory: 2Gi
|
||||||
|
scopes:
|
||||||
|
- NotBestEffort
|
|
@ -0,0 +1,9 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ResourceQuota
|
||||||
|
metadata:
|
||||||
|
name: object-counts
|
||||||
|
spec:
|
||||||
|
hard:
|
||||||
|
persistentvolumeclaims: "2"
|
||||||
|
services.loadbalancers: "2"
|
||||||
|
services.nodeports: "0"
|
|
@ -1,14 +0,0 @@
|
||||||
apiVersion: v1
|
|
||||||
kind: ResourceQuota
|
|
||||||
metadata:
|
|
||||||
name: quota
|
|
||||||
spec:
|
|
||||||
hard:
|
|
||||||
cpu: "20"
|
|
||||||
memory: 1Gi
|
|
||||||
persistentvolumeclaims: "10"
|
|
||||||
pods: "10"
|
|
||||||
replicationcontrollers: "20"
|
|
||||||
resourcequotas: "1"
|
|
||||||
secrets: "10"
|
|
||||||
services: "5"
|
|
|
@ -1,12 +1,30 @@
|
||||||
---
|
---
|
||||||
---
|
---
|
||||||
|
|
||||||
This example demonstrates how [resource quota](/docs/admin/admission-controllers/#resourcequota) and
|
This example demonstrates a typical setup to control for resource usage in a namespace.
|
||||||
[limitsranger](/docs/admin/admission-controllers/#limitranger) can be applied to a Kubernetes namespace.
|
|
||||||
See [ResourceQuota design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/admission_control_resource_quota.md) for more information.
|
It demonstrates using the following resources:
|
||||||
|
|
||||||
|
* [Namespace](/docs/admin/namespaces)
|
||||||
|
* [Resource Quota](/docs/admin/resourcequota/)
|
||||||
|
* [Limit Range](/docs/admin/limitrange/)
|
||||||
|
|
||||||
This example assumes you have a functional Kubernetes setup.
|
This example assumes you have a functional Kubernetes setup.
|
||||||
|
|
||||||
|
## Scenario
|
||||||
|
|
||||||
|
The cluster-admin is operating a cluster on behalf of a user population and the cluster-admin
|
||||||
|
wants to control the amount of resources that can be consumed in a particular namespace to promote
|
||||||
|
fair sharing of the cluster and control cost.
|
||||||
|
|
||||||
|
The cluster-admin has the following goals:
|
||||||
|
|
||||||
|
* Limit the amount of compute resource for running pods
|
||||||
|
* Limit the number of persistent volume claims to control access to storage
|
||||||
|
* Limit the number of load balancers to control cost
|
||||||
|
* Prevent the use of node ports to preserve scarce resources
|
||||||
|
* Provide default compute resource requests to enable better scheduling decisions
|
||||||
|
|
||||||
## Step 1: Create a namespace
|
## Step 1: Create a namespace
|
||||||
|
|
||||||
This example will work in a custom namespace to demonstrate the concepts involved.
|
This example will work in a custom namespace to demonstrate the concepts involved.
|
||||||
|
@ -14,104 +32,140 @@ This example will work in a custom namespace to demonstrate the concepts involve
|
||||||
Let's create a new namespace called quota-example:
|
Let's create a new namespace called quota-example:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl create namespace quota-example
|
$ kubectl create -f docs/admin/resourcequota/namespace.yaml
|
||||||
namespace "quota-example" created
|
namespace "quota-example" created
|
||||||
```
|
|
||||||
|
|
||||||
Note that `kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ kubectl get namespaces
|
$ kubectl get namespaces
|
||||||
NAME STATUS AGE
|
NAME STATUS AGE
|
||||||
default Active 50m
|
default Active 2m
|
||||||
quota-example Active 2s
|
kube-system Active 2m
|
||||||
|
quota-example Active 39s
|
||||||
```
|
```
|
||||||
|
|
||||||
## Step 2: Apply a quota to the namespace
|
## Step 2: Apply an object-count quota to the namespace
|
||||||
|
|
||||||
By default, a pod will run with unbounded CPU and memory requests/limits. This means that any pod in the
|
The cluster-admin wants to control the following resources:
|
||||||
system will be able to consume as much CPU and memory on the node that executes the pod.
|
|
||||||
|
|
||||||
Users may want to restrict how much of the cluster resources a given namespace may consume
|
* persistent volume claims
|
||||||
across all of its pods in order to manage cluster usage. To do this, a user applies a quota to
|
* load balancers
|
||||||
a namespace. A quota lets the user set hard limits on the total amount of node resources (cpu, memory)
|
* node ports
|
||||||
and API resources (pods, services, etc.) that a namespace may consume. In term of resources, Kubernetes
|
|
||||||
checks the total resource *requests*, not resource *limits* of all containers/pods in the namespace.
|
|
||||||
|
|
||||||
Let's create a simple quota in our namespace:
|
Let's create a simple quota that controls object counts for those resource types in this namespace.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl create -f docs/admin/resourcequota/quota.yaml --namespace=quota-example
|
$ kubectl create -f docs/admin/resourcequota/object-counts.yaml --namespace=quota-example
|
||||||
resourcequota "quota" created
|
resourcequota "object-counts" created
|
||||||
```
|
```
|
||||||
|
|
||||||
Once your quota is applied to a namespace, the system will restrict any creation of content
|
The quota system will observe that a quota has been created, and will calculate consumption
|
||||||
in the namespace until the quota usage has been calculated. This should happen quickly.
|
in the namespace in response. This should happen quickly.
|
||||||
|
|
||||||
You can describe your current quota usage to see what resources are being consumed in your
|
Let's describe the quota to see what is currently being consumed in this namespace:
|
||||||
namespace.
|
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl describe quota quota --namespace=quota-example
|
$ kubectl describe quota object-counts --namespace=quota-example
|
||||||
Name: quota
|
Name: object-counts
|
||||||
Namespace: quota-example
|
Namespace: quota-example
|
||||||
Resource Used Hard
|
Resource Used Hard
|
||||||
-------- ---- ----
|
-------- ---- ----
|
||||||
cpu 0 20
|
persistentvolumeclaims 0 2
|
||||||
memory 0 1Gi
|
services.loadbalancers 0 2
|
||||||
persistentvolumeclaims 0 10
|
services.nodeports 0 0
|
||||||
pods 0 10
|
|
||||||
replicationcontrollers 0 20
|
|
||||||
resourcequotas 1 1
|
|
||||||
secrets 1 10
|
|
||||||
services 0 5
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Step 3: Applying default resource requests and limits
|
The quota system will now prevent users from creating more than the specified amount for each resource.
|
||||||
|
|
||||||
|
## Step 3: Apply a compute-resource quota to the namespace
|
||||||
|
|
||||||
|
To limit the amount of compute resource that can be consumed in this namespace,
|
||||||
|
let's create a quota that tracks compute resources.
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl create -f docs/admin/resourcequota/compute-resources.yaml --namespace=quota-example
|
||||||
|
resourcequota "compute-resources" created
|
||||||
|
```
|
||||||
|
|
||||||
|
Let's describe the quota to see what is currently being consumed in this namespace:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl describe quota compute-resources --namespace=quota-example
|
||||||
|
Name: compute-resources
|
||||||
|
Namespace: quota-example
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
limits.cpu 0 2
|
||||||
|
limits.memory 0 2Gi
|
||||||
|
pods 0 4
|
||||||
|
requests.cpu 0 1
|
||||||
|
requests.memory 0 1Gi
|
||||||
|
```
|
||||||
|
|
||||||
|
The quota system will now prevent the namespace from having more than 4 non-terminal pods. In
|
||||||
|
addition, it will enforce that each container in a pod makes a `request` and defines a `limit` for
|
||||||
|
`cpu` and `memory`.
|
||||||
|
|
||||||
|
## Step 4: Applying default resource requests and limits
|
||||||
|
|
||||||
Pod authors rarely specify resource requests and limits for their pods.
|
Pod authors rarely specify resource requests and limits for their pods.
|
||||||
|
|
||||||
Since we applied a quota to our project, let's see what happens when an end-user creates a pod that has unbounded
|
Since we applied a quota to our project, let's see what happens when an end-user creates a pod that has unbounded
|
||||||
cpu and memory by creating an nginx container.
|
cpu and memory by creating an nginx container.
|
||||||
|
|
||||||
To demonstrate, lets create a Deployment that runs nginx:
|
To demonstrate, lets create a deployment that runs nginx:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl run nginx --image=nginx --replicas=1 --namespace=quota-example
|
$ kubectl run nginx --image=nginx --replicas=1 --namespace=quota-example
|
||||||
deployment "nginx" created
|
deployment "nginx" created
|
||||||
```
|
```
|
||||||
|
|
||||||
This creates a Deployment "nginx" with its underlying resource, a ReplicaSet, which handles the creation and deletion of Pod replicas. Now let's look at the pods that were created.
|
Now let's look at the pods that were created.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl get pods --namespace=quota-example
|
$ kubectl get pods --namespace=quota-example
|
||||||
NAME READY STATUS RESTARTS AGE
|
|
||||||
```
|
```
|
||||||
|
|
||||||
What happened? I have no pods! Let's describe the ReplicaSet managed by the nginx Deployment to get a view of what is happening.
|
What happened? I have no pods! Let's describe the deployment to get a view of what is happening.
|
||||||
Note that `kubectl describe rs` works only on kubernetes cluster >= v1.2. If you are running older versions, use `kubectl describe rc` instead.
|
|
||||||
If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/user-guide/kubectl/kubectl_run/) for more details.
|
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl describe rs -l run=nginx --namespace=quota-example
|
$ kubectl describe deployment nginx --namespace=quota-example
|
||||||
Name: nginx-2040093540
|
Name: nginx
|
||||||
|
Namespace: quota-example
|
||||||
|
CreationTimestamp: Mon, 06 Jun 2016 16:11:37 -0400
|
||||||
|
Labels: run=nginx
|
||||||
|
Selector: run=nginx
|
||||||
|
Replicas: 0 updated | 1 total | 0 available | 1 unavailable
|
||||||
|
StrategyType: RollingUpdate
|
||||||
|
MinReadySeconds: 0
|
||||||
|
RollingUpdateStrategy: 1 max unavailable, 1 max surge
|
||||||
|
OldReplicaSets: <none>
|
||||||
|
NewReplicaSet: nginx-3137573019 (0/1 replicas created)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
A deployment created a corresponding replica set and attempted to size it to create a single pod.
|
||||||
|
|
||||||
|
Let's look at the replica set to get more detail.
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl describe rs nginx-3137573019 --namespace=quota-example
|
||||||
|
Name: nginx-3137573019
|
||||||
Namespace: quota-example
|
Namespace: quota-example
|
||||||
Image(s): nginx
|
Image(s): nginx
|
||||||
Selector: pod-template-hash=2040093540,run=nginx
|
Selector: pod-template-hash=3137573019,run=nginx
|
||||||
Labels: pod-template-hash=2040093540,run=nginx
|
Labels: pod-template-hash=3137573019
|
||||||
|
run=nginx
|
||||||
Replicas: 0 current / 1 desired
|
Replicas: 0 current / 1 desired
|
||||||
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
|
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
|
||||||
No volumes.
|
No volumes.
|
||||||
Events:
|
Events:
|
||||||
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
|
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
|
||||||
--------- -------- ----- ---- ------------- -------- ------ -------
|
--------- -------- ----- ---- ------------- -------- ------ -------
|
||||||
48s 26s 4 {replicaset-controller } Warning FailedCreate Error creating: pods "nginx-2040093540-" is forbidden: Failed quota: quota: must specify cpu,memory
|
4m 7s 11 {replicaset-controller } Warning FailedCreate Error creating: pods "nginx-3137573019-" is forbidden: Failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
|
||||||
```
|
```
|
||||||
|
|
||||||
The Kubernetes API server is rejecting the ReplicaSet requests to create a pod because our pods
|
The Kubernetes API server is rejecting the replica set requests to create a pod because our pods
|
||||||
do not specify any memory usage *request*.
|
do not specify `requests` or `limits` for `cpu` and `memory`.
|
||||||
|
|
||||||
So let's set some default values for the amount of cpu and memory a pod can consume:
|
So let's set some default values for the amount of `cpu` and `memory` a pod can consume:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl create -f docs/admin/resourcequota/limits.yaml --namespace=quota-example
|
$ kubectl create -f docs/admin/resourcequota/limits.yaml --namespace=quota-example
|
||||||
|
@ -121,44 +175,187 @@ Name: limits
|
||||||
Namespace: quota-example
|
Namespace: quota-example
|
||||||
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
|
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
|
||||||
---- -------- --- --- --------------- ------------- -----------------------
|
---- -------- --- --- --------------- ------------- -----------------------
|
||||||
Container cpu - - 100m 200m -
|
|
||||||
Container memory - - 256Mi 512Mi -
|
Container memory - - 256Mi 512Mi -
|
||||||
|
Container cpu - - 100m 200m -
|
||||||
```
|
```
|
||||||
|
|
||||||
Now any time a pod is created in this namespace, if it has not specified any resource request/limit, the default
|
If the Kubernetes API server observes a request to create a pod in this namespace, and the containers
|
||||||
amount of cpu and memory per container will be applied, and the request will be used as part of admission control.
|
in that pod do not make any compute resource requests, a default request and default limit will be applied
|
||||||
|
as part of admission control.
|
||||||
|
|
||||||
Now that we have applied default resource *request* for our namespace, our Deployment should be able to
|
In this example, each pod created will have compute resources equivalent to the following:
|
||||||
create its pods.
|
|
||||||
|
```shell
|
||||||
|
$ kubectl run nginx \
|
||||||
|
--image=nginx \
|
||||||
|
--replicas=1 \
|
||||||
|
--requests=cpu=100m,memory=256Mi \
|
||||||
|
--limits=cpu=200m,memory=512Mi \
|
||||||
|
--namespace=quota-example
|
||||||
|
```
|
||||||
|
|
||||||
|
Now that we have applied default compute resources for our namespace, our replica set should be able to create
|
||||||
|
its pods.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl get pods --namespace=quota-example
|
$ kubectl get pods --namespace=quota-example
|
||||||
NAME READY STATUS RESTARTS AGE
|
NAME READY STATUS RESTARTS AGE
|
||||||
nginx-2040093540-miohp 1/1 Running 0 5s
|
nginx-3137573019-fvrig 1/1 Running 0 6m
|
||||||
```
|
```
|
||||||
|
|
||||||
And if we print out our quota usage in the namespace:
|
And if we print out our quota usage in the namespace:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ kubectl describe quota quota --namespace=quota-example
|
$ kubectl describe quota --namespace=quota-example
|
||||||
Name: quota
|
Name: compute-resources
|
||||||
Namespace: quota-example
|
Namespace: quota-example
|
||||||
Resource Used Hard
|
Resource Used Hard
|
||||||
-------- ---- ----
|
-------- ---- ----
|
||||||
cpu 100m 20
|
limits.cpu 200m 2
|
||||||
memory 256Mi 1Gi
|
limits.memory 512Mi 2Gi
|
||||||
persistentvolumeclaims 0 10
|
pods 1 4
|
||||||
pods 1 10
|
requests.cpu 100m 1
|
||||||
replicationcontrollers 1 20
|
requests.memory 256Mi 1Gi
|
||||||
resourcequotas 1 1
|
|
||||||
secrets 1 10
|
|
||||||
services 0 5
|
Name: object-counts
|
||||||
|
Namespace: quota-example
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
persistentvolumeclaims 0 2
|
||||||
|
services.loadbalancers 0 2
|
||||||
|
services.nodeports 0 0
|
||||||
```
|
```
|
||||||
|
|
||||||
You can now see the pod that was created is consuming explicit amounts of resources (specified by resource *request*), and the usage is being tracked by the Kubernetes system properly.
|
As you can see, the pod that was created is consuming explict amounts of compute resources, and the usage is being
|
||||||
|
tracked by Kubernetes properly.
|
||||||
|
|
||||||
|
## Step 5: Advanced quota scopes
|
||||||
|
|
||||||
|
Let's imagine you did not want to specify default compute resource consumption in your namespace.
|
||||||
|
|
||||||
|
Instead, you want to let users run a specific number of `BestEffort` pods in their namespace to take
|
||||||
|
advantage of slack compute resources, and then require that users make an explicit resource request for
|
||||||
|
pods that require a higher quality of service.
|
||||||
|
|
||||||
|
Let's create a new namespace with two quotas to demonstrate this behavior:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl create namespace quota-scopes
|
||||||
|
namespace "quota-scopes" created
|
||||||
|
$ kubectl create -f docs/admin/resourcequota/best-effort.yaml --namespace=quota-scopes
|
||||||
|
resourcequota "best-effort" created
|
||||||
|
$ kubectl create -f docs/admin/resourcequota/not-best-effort.yaml --namespace=quota-scopes
|
||||||
|
resourcequota "not-best-effort" created
|
||||||
|
$ kubectl describe quota --namespace=quota-scopes
|
||||||
|
Name: best-effort
|
||||||
|
Namespace: quota-scopes
|
||||||
|
Scopes: BestEffort
|
||||||
|
* Matches all pods that have best effort quality of service.
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
pods 0 10
|
||||||
|
|
||||||
|
|
||||||
|
Name: not-best-effort
|
||||||
|
Namespace: quota-scopes
|
||||||
|
Scopes: NotBestEffort
|
||||||
|
* Matches all pods that do not have best effort quality of service.
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
limits.cpu 0 2
|
||||||
|
limits.memory 0 2Gi
|
||||||
|
pods 0 4
|
||||||
|
requests.cpu 0 1
|
||||||
|
requests.memory 0 1Gi
|
||||||
|
```
|
||||||
|
|
||||||
|
In this scenario, a pod that makes no compute resource requests will be tracked by the `best-effort` quota.
|
||||||
|
|
||||||
|
A pod that does make compute resource requests will be tracked by the `not-best-effort` quota.
|
||||||
|
|
||||||
|
Let's demonstrate this by creating two deployments:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl run best-effort-nginx --image=nginx --replicas=8 --namespace=quota-scopes
|
||||||
|
deployment "best-effort-nginx" created
|
||||||
|
$ kubectl run not-best-effort-nginx \
|
||||||
|
--image=nginx \
|
||||||
|
--replicas=2 \
|
||||||
|
--requests=cpu=100m,memory=256Mi \
|
||||||
|
--limits=cpu=200m,memory=512Mi \
|
||||||
|
--namespace=quota-scopes
|
||||||
|
deployment "not-best-effort-nginx" created
|
||||||
|
```
|
||||||
|
|
||||||
|
Even though no default limits were specified, the `best-effort-nginx` deployment will create
|
||||||
|
all 8 pods. This is because it is tracked by the `best-effort` quota, and the `not-best-effort`
|
||||||
|
quota will just ignore it. The `not-best-effort` quota will track the `not-best-effort-nginx`
|
||||||
|
deployment since it creates pods with `Burstable` quality of service.
|
||||||
|
|
||||||
|
Let's list the pods in the namespace:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl get pods --namespace=quota-scopes
|
||||||
|
NAME READY STATUS RESTARTS AGE
|
||||||
|
best-effort-nginx-3488455095-2qb41 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-3go7n 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-9o2xg 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-eyg40 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-gcs3v 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-rq8p1 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-udhhd 1/1 Running 0 51s
|
||||||
|
best-effort-nginx-3488455095-zmk12 1/1 Running 0 51s
|
||||||
|
not-best-effort-nginx-2204666826-7sl61 1/1 Running 0 23s
|
||||||
|
not-best-effort-nginx-2204666826-ke746 1/1 Running 0 23s
|
||||||
|
```
|
||||||
|
|
||||||
|
As you can see, all 10 pods have been allowed to be created.
|
||||||
|
|
||||||
|
Let's describe current quota usage in the namespace:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ kubectl describe quota --namespace=quota-scopes
|
||||||
|
Name: best-effort
|
||||||
|
Namespace: quota-scopes
|
||||||
|
Scopes: BestEffort
|
||||||
|
* Matches all pods that have best effort quality of service.
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
pods 8 10
|
||||||
|
|
||||||
|
|
||||||
|
Name: not-best-effort
|
||||||
|
Namespace: quota-scopes
|
||||||
|
Scopes: NotBestEffort
|
||||||
|
* Matches all pods that do not have best effort quality of service.
|
||||||
|
Resource Used Hard
|
||||||
|
-------- ---- ----
|
||||||
|
limits.cpu 400m 2
|
||||||
|
limits.memory 1Gi 2Gi
|
||||||
|
pods 2 4
|
||||||
|
requests.cpu 200m 1
|
||||||
|
requests.memory 512Mi 1Gi
|
||||||
|
```
|
||||||
|
|
||||||
|
As you can see, the `best-effort` quota has tracked the usage for the 8 pods we created in
|
||||||
|
the `best-effort-nginx` deployment, and the `not-best-effort` quota has tracked the usage for
|
||||||
|
the 2 pods we created in the `not-best-effort-nginx` quota.
|
||||||
|
|
||||||
|
Scopes provide a mechanism to subdivide the set of resources that are tracked by
|
||||||
|
any quota document to allow greater flexibility in how operators deploy and track resource
|
||||||
|
consumption.
|
||||||
|
|
||||||
|
In addition to `BestEffort` and `NotBestEffort` scopes, there are scopes to restrict
|
||||||
|
long-running versus time-bound pods. The `Terminating` scope will match any pod
|
||||||
|
where `spec.activeDeadlineSeconds is not nil`. The `NotTerminating` scope will match any pod
|
||||||
|
where `spec.activeDeadlineSeconds is nil`. These scopes allow you to quota pods based on their
|
||||||
|
anticipated permanence on a node in your cluster.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
Actions that consume node resources for cpu and memory can be subject to hard quota limits defined by the namespace quota. The resource consumption is measured by resource *request* in pod specification.
|
Actions that consume node resources for cpu and memory can be subject to hard quota limits defined by the namespace quota.
|
||||||
|
|
||||||
Any action that consumes those resources can be tweaked, or can pick up namespace level defaults to meet your end goal.
|
Any action that consumes those resources can be tweaked, or can pick up namespace level defaults to meet your end goal.
|
||||||
|
|
||||||
|
Quota can be apportioned based on quality of service and anticipated permanence on a node in your cluster.
|
Loading…
Reference in New Issue