574 lines
17 KiB
Markdown
574 lines
17 KiB
Markdown
---
|
|
title: Extend the Kubernetes API with CustomResourceDefinitions
|
|
reviewers:
|
|
- deads2k
|
|
- enisoc
|
|
content_template: templates/task
|
|
weight: 20
|
|
---
|
|
|
|
{{% capture overview %}}
|
|
This page shows how to install a
|
|
[custom resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
|
|
into the Kubernetes API by creating a
|
|
[CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
|
|
{{% /capture %}}
|
|
|
|
{{% capture prerequisites %}}
|
|
|
|
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
|
|
|
* Make sure your Kubernetes cluster has a master version of 1.7.0 or higher.
|
|
|
|
* Read about [custom resources](/docs/concepts/api-extension/custom-resources/).
|
|
|
|
{{% /capture %}}
|
|
|
|
{{% capture steps %}}
|
|
## Create a CustomResourceDefinition
|
|
|
|
When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server
|
|
creates a new RESTful resource path for each version you specify. The CRD can be
|
|
either namespaced or cluster-scoped, as specified in the CRD's `scope` field. As
|
|
with existing built-in objects, deleting a namespace deletes all custom objects
|
|
in that namespace. CustomResourceDefinitions themselves are non-namespaced and
|
|
are available to all namespaces.
|
|
|
|
For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: apiextensions.k8s.io/v1beta1
|
|
kind: CustomResourceDefinition
|
|
metadata:
|
|
# name must match the spec fields below, and be in the form: <plural>.<group>
|
|
name: crontabs.stable.example.com
|
|
spec:
|
|
# group name to use for REST API: /apis/<group>/<version>
|
|
group: stable.example.com
|
|
# list of versions supported by this CustomResourceDefinition
|
|
versions:
|
|
- name: v1
|
|
# Each version can be enabled/disabled by Served flag.
|
|
served: true
|
|
# One and only one version must be marked as the storage version.
|
|
storage: true
|
|
# either Namespaced or Cluster
|
|
scope: Namespaced
|
|
names:
|
|
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
|
|
plural: crontabs
|
|
# singular name to be used as an alias on the CLI and for display
|
|
singular: crontab
|
|
# kind is normally the CamelCased singular type. Your resource manifests use this.
|
|
kind: CronTab
|
|
# shortNames allow shorter string to match your resource on the CLI
|
|
shortNames:
|
|
- ct
|
|
```
|
|
|
|
And create it:
|
|
|
|
```shell
|
|
kubectl create -f resourcedefinition.yaml
|
|
```
|
|
|
|
Then a new namespaced RESTful API endpoint is created at:
|
|
|
|
```
|
|
/apis/stable.example.com/v1/namespaces/*/crontabs/...
|
|
```
|
|
|
|
This endpoint URL can then be used to create and manage custom objects.
|
|
The `kind` of these objects will be `CronTab` from the spec of the
|
|
CustomResourceDefinition object you created above.
|
|
|
|
It might take a few seconds for the endpoint to be created.
|
|
You can watch the `Established` condition of your CustomResourceDefinition
|
|
to be true or watch the discovery information of the API server for your
|
|
resource to show up.
|
|
|
|
## Create custom objects
|
|
|
|
After the CustomResourceDefinition object has been created, you can create
|
|
custom objects. Custom objects can contain custom fields. These fields can
|
|
contain arbitrary JSON.
|
|
In the following example, the `cronSpec` and `image` custom fields are set in a
|
|
custom object of kind `CronTab`. The kind `CronTab` comes from the spec of the
|
|
CustomResourceDefinition object you created above.
|
|
|
|
If you save the following YAML to `my-crontab.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: "stable.example.com/v1"
|
|
kind: CronTab
|
|
metadata:
|
|
name: my-new-cron-object
|
|
spec:
|
|
cronSpec: "* * * * */5"
|
|
image: my-awesome-cron-image
|
|
```
|
|
|
|
and create it:
|
|
|
|
```shell
|
|
kubectl create -f my-crontab.yaml
|
|
```
|
|
|
|
You can then manage your CronTab objects using kubectl. For example:
|
|
|
|
```shell
|
|
kubectl get crontab
|
|
```
|
|
|
|
Should print a list like this:
|
|
|
|
```console
|
|
NAME AGE
|
|
my-new-cron-object 6s
|
|
```
|
|
|
|
Resource names are not case-sensitive when using kubectl, and you can use either
|
|
the singular or plural forms defined in the CRD, as well as any short names.
|
|
|
|
You can also view the raw YAML data:
|
|
|
|
```shell
|
|
kubectl get ct -o yaml
|
|
```
|
|
|
|
You should see that it contains the custom `cronSpec` and `image` fields
|
|
from the yaml you used to create it:
|
|
|
|
```console
|
|
apiVersion: v1
|
|
items:
|
|
- apiVersion: stable.example.com/v1
|
|
kind: CronTab
|
|
metadata:
|
|
clusterName: ""
|
|
creationTimestamp: 2017-05-31T12:56:35Z
|
|
deletionGracePeriodSeconds: null
|
|
deletionTimestamp: null
|
|
name: my-new-cron-object
|
|
namespace: default
|
|
resourceVersion: "285"
|
|
selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
|
|
uid: 9423255b-4600-11e7-af6a-28d2447dc82b
|
|
spec:
|
|
cronSpec: '* * * * */5'
|
|
image: my-awesome-cron-image
|
|
kind: List
|
|
metadata:
|
|
resourceVersion: ""
|
|
selfLink: ""
|
|
```
|
|
|
|
## Delete a CustomResourceDefinition
|
|
|
|
When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint
|
|
and **delete all custom objects stored in it**.
|
|
|
|
```shell
|
|
kubectl delete -f resourcedefinition.yaml
|
|
kubectl get crontabs
|
|
```
|
|
|
|
```console
|
|
Error from server (NotFound): Unable to list "crontabs": the server could not find the requested resource (get crontabs.stable.example.com)
|
|
```
|
|
|
|
If you later recreate the same CustomResourceDefinition, it will start out empty.
|
|
|
|
## Serving multiple versions of a CRD
|
|
|
|
See [Custom resource definition versioning](custom-resource-definition-versioning)
|
|
for more information about serving multiple versions of your
|
|
CustomResourceDefinition and migrating your objects from one version to another.
|
|
|
|
{{% /capture %}}
|
|
|
|
{{% capture discussion %}}
|
|
## Advanced topics
|
|
|
|
### Finalizers
|
|
|
|
*Finalizers* allow controllers to implement asynchronous pre-delete hooks.
|
|
Custom objects support finalizers just like built-in objects.
|
|
|
|
You can add a finalizer to a custom object like this:
|
|
|
|
```yaml
|
|
apiVersion: "stable.example.com/v1"
|
|
kind: CronTab
|
|
metadata:
|
|
finalizers:
|
|
- finalizer.stable.example.com
|
|
```
|
|
|
|
Finalizers are arbitrary string values, that when present ensure that a hard delete
|
|
of a resource is not possible while they exist.
|
|
|
|
The first delete request on an object with finalizers merely sets a value for the
|
|
`metadata.deletionTimestamp` field instead of deleting it. Once this value is set,
|
|
entries in the `finalizer` list can only be removed.
|
|
|
|
This triggers controllers watching the object to execute any finalizers they handle.
|
|
This will be represented via polling update requests for that
|
|
object, until all finalizers have been removed and the resource is deleted.
|
|
|
|
The time period of polling update can be controlled by `metadata.deletionGracePeriodSeconds`.
|
|
|
|
It is the responsibility of each controller to removes its finalizer from the list.
|
|
|
|
Kubernetes will only finally delete the object if the list of finalizers is empty,
|
|
meaning all finalizers are done.
|
|
|
|
### Validation
|
|
|
|
Validation of custom objects is possible via
|
|
[OpenAPI v3 schema](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject).
|
|
Additionally, the following restrictions are applied to the schema:
|
|
|
|
- The fields `default`, `nullable`, `discriminator`, `readOnly`, `writeOnly`, `xml`,
|
|
`deprecated` and `$ref` cannot be set.
|
|
- The field `uniqueItems` cannot be set to true.
|
|
- The field `additionalProperties` cannot be set to false.
|
|
|
|
This feature is __beta__ in v1.9.
|
|
You can disable this feature using the `CustomResourceValidation` feature gate on
|
|
the [kube-apiserver](/docs/admin/kube-apiserver):
|
|
|
|
```
|
|
--feature-gates=CustomResourceValidation=false
|
|
```
|
|
|
|
The schema is defined in the CustomResourceDefinition. In the following example, the
|
|
CustomResourceDefinition applies the following validations on the custom object:
|
|
|
|
- `spec.cronSpec` must be a string and must be of the form described by the regular expression.
|
|
- `spec.replicas` must be an integer and must have a minimum value of 1 and a maximum value of 10.
|
|
|
|
Save the CustomResourceDefinition to `resourcedefinition.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: apiextensions.k8s.io/v1beta1
|
|
kind: CustomResourceDefinition
|
|
metadata:
|
|
name: crontabs.stable.example.com
|
|
spec:
|
|
group: stable.example.com
|
|
versions:
|
|
- name: v1
|
|
served: true
|
|
storage: true
|
|
version: v1
|
|
scope: Namespaced
|
|
names:
|
|
plural: crontabs
|
|
singular: crontab
|
|
kind: CronTab
|
|
shortNames:
|
|
- ct
|
|
validation:
|
|
# openAPIV3Schema is the schema for validating custom objects.
|
|
openAPIV3Schema:
|
|
properties:
|
|
spec:
|
|
properties:
|
|
cronSpec:
|
|
type: string
|
|
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
|
|
replicas:
|
|
type: integer
|
|
minimum: 1
|
|
maximum: 10
|
|
```
|
|
|
|
And create it:
|
|
|
|
```shell
|
|
kubectl create -f resourcedefinition.yaml
|
|
```
|
|
|
|
A request to create a custom object of kind `CronTab` will be rejected if there are invalid values in its fields.
|
|
In the following example, the custom object contains fields with invalid values:
|
|
|
|
- `spec.cronSpec` does not match the regular expression.
|
|
- `spec.replicas` is greater than 10.
|
|
|
|
If you save the following YAML to `my-crontab.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: "stable.example.com/v1"
|
|
kind: CronTab
|
|
metadata:
|
|
name: my-new-cron-object
|
|
spec:
|
|
cronSpec: "* * * *"
|
|
image: my-awesome-cron-image
|
|
replicas: 15
|
|
```
|
|
|
|
and create it:
|
|
|
|
```shell
|
|
kubectl create -f my-crontab.yaml
|
|
```
|
|
|
|
you will get an error:
|
|
|
|
```console
|
|
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
|
|
validation failure list:
|
|
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
|
|
spec.replicas in body should be less than or equal to 10
|
|
```
|
|
|
|
If the fields contain valid values, the object creation request is accepted.
|
|
|
|
Save the following YAML to `my-crontab.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: "stable.example.com/v1"
|
|
kind: CronTab
|
|
metadata:
|
|
name: my-new-cron-object
|
|
spec:
|
|
cronSpec: "* * * * */5"
|
|
image: my-awesome-cron-image
|
|
replicas: 5
|
|
```
|
|
|
|
And create it:
|
|
|
|
```shell
|
|
kubectl create -f my-crontab.yaml
|
|
crontab "my-new-cron-object" created
|
|
```
|
|
|
|
### Subresources
|
|
|
|
Custom resources support `/status` and `/scale` subresources.
|
|
This feature is __beta__ in v1.11 and enabled by default.
|
|
|
|
You can disable this feature using the `CustomResourceSubresources` feature gate on
|
|
the [kube-apiserver](/docs/admin/kube-apiserver):
|
|
|
|
```
|
|
--feature-gates=CustomResourceSubresources=false
|
|
```
|
|
|
|
The status and scale subresources can be optionally enabled by
|
|
defining them in the CustomResourceDefinition.
|
|
|
|
#### Status subresource
|
|
|
|
When the status subresource is enabled, the `/status` subresource for the custom resource is exposed.
|
|
|
|
- The status and the spec stanzas are represented by the `.status` and `.spec` JSONPaths respectively inside of a custom resource.
|
|
- `PUT` requests to the `/status` subresource take a custom resource object and ignore changes to anything except the status stanza.
|
|
- `PUT` requests to the `/status` subresource only validate the status stanza of the custom resource.
|
|
- `PUT`/`POST`/`PATCH` requests to the custom resource ignore changes to the status stanza.
|
|
- Any changes to the spec stanza increments the value at `.metadata.generation`.
|
|
- `properties`, `required` and `description` are the only constructs allowed in the root of the CRD OpenAPI validation schema.
|
|
|
|
#### Scale subresource
|
|
|
|
When the scale subresource is enabled, the `/scale` subresource for the custom resource is exposed.
|
|
The `autoscaling/v1.Scale` object is sent as the payload for `/scale`.
|
|
|
|
To enable the scale subresource, the following values are defined in the CustomResourceDefinition.
|
|
|
|
- `SpecReplicasPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Spec.Replicas`.
|
|
|
|
- It is a required value.
|
|
- Only JSONPaths under `.spec` and with the dot notation are allowed.
|
|
- If there is no value under the `SpecReplicasPath` in the custom resource,
|
|
the `/scale` subresource will return an error on GET.
|
|
|
|
- `StatusReplicasPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Status.Replicas`.
|
|
|
|
- It is a required value.
|
|
- Only JSONPaths under `.status` and with the dotation are allowed.
|
|
- If there is no value under the `StatusReplicasPath` in the custom resource,
|
|
the status replica value in the `/scale` subresource will default to 0.
|
|
|
|
- `LabelSelectorPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Status.Selector`.
|
|
|
|
- It is an optional value.
|
|
- It must be set to work with HPA.
|
|
- Only JSONPaths under `.status` and with the dotation are allowed.
|
|
- If there is no value under the `LabelSelectorPath` in the custom resource,
|
|
the status selector value in the `/scale` subresource will default to the empty string.
|
|
|
|
In the following example, both status and scale subresources are enabled.
|
|
|
|
Save the CustomResourceDefinition to `resourcedefinition.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: apiextensions.k8s.io/v1beta1
|
|
kind: CustomResourceDefinition
|
|
metadata:
|
|
name: crontabs.stable.example.com
|
|
spec:
|
|
group: stable.example.com
|
|
versions:
|
|
- name: v1
|
|
served: true
|
|
storage: true
|
|
scope: Namespaced
|
|
names:
|
|
plural: crontabs
|
|
singular: crontab
|
|
kind: CronTab
|
|
shortNames:
|
|
- ct
|
|
# subresources describes the subresources for custom resources.
|
|
subresources:
|
|
# status enables the status subresource.
|
|
status: {}
|
|
# scale enables the scale subresource.
|
|
scale:
|
|
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
|
|
specReplicasPath: .spec.replicas
|
|
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
|
|
statusReplicasPath: .status.replicas
|
|
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
|
|
labelSelectorPath: .status.labelSelector
|
|
```
|
|
|
|
And create it:
|
|
|
|
```shell
|
|
kubectl create -f resourcedefinition.yaml
|
|
```
|
|
|
|
After the CustomResourceDefinition object has been created, you can create custom objects.
|
|
|
|
If you save the following YAML to `my-crontab.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: "stable.example.com/v1"
|
|
kind: CronTab
|
|
metadata:
|
|
name: my-new-cron-object
|
|
spec:
|
|
cronSpec: "* * * * */5"
|
|
image: my-awesome-cron-image
|
|
replicas: 3
|
|
```
|
|
|
|
and create it:
|
|
|
|
```shell
|
|
kubectl create -f my-crontab.yaml
|
|
```
|
|
|
|
Then new namespaced RESTful API endpoints are created at:
|
|
|
|
```
|
|
/apis/stable.example.com/v1/namespaces/*/crontabs/status
|
|
```
|
|
|
|
and
|
|
|
|
```
|
|
/apis/stable.example.com/v1/namespaces/*/crontabs/scale
|
|
```
|
|
|
|
A custom resource can be scaled using the `kubectl scale` command.
|
|
For example, the following command sets `.spec.replicas` of the
|
|
custom resource created above to 5:
|
|
|
|
```shell
|
|
kubectl scale --replicas=5 crontabs/my-new-cron-object
|
|
crontabs "my-new-cron-object" scaled
|
|
|
|
kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
|
|
5
|
|
```
|
|
|
|
### Categories
|
|
|
|
Categories is a list of grouped resources the custom resource belongs to (eg. `all`).
|
|
You can use `kubectl get <category-name>` to list the resources belonging to the category.
|
|
This feature is __beta__ and available for custom resources from v1.10.
|
|
|
|
The following example adds `all` in the list of categories in the CustomResourceDefinition
|
|
and illustrates how to output the custom resource using `kubectl get all`.
|
|
|
|
Save the following CustomResourceDefinition to `resourcedefinition.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: apiextensions.k8s.io/v1beta1
|
|
kind: CustomResourceDefinition
|
|
metadata:
|
|
name: crontabs.stable.example.com
|
|
spec:
|
|
group: stable.example.com
|
|
versions:
|
|
- name: v1
|
|
served: true
|
|
storage: true
|
|
scope: Namespaced
|
|
names:
|
|
plural: crontabs
|
|
singular: crontab
|
|
kind: CronTab
|
|
shortNames:
|
|
- ct
|
|
# categories is a list of grouped resources the custom resource belongs to.
|
|
categories:
|
|
- all
|
|
```
|
|
|
|
And create it:
|
|
|
|
```shell
|
|
kubectl create -f resourcedefinition.yaml
|
|
```
|
|
|
|
After the CustomResourceDefinition object has been created, you can create custom objects.
|
|
|
|
Save the following YAML to `my-crontab.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: "stable.example.com/v1"
|
|
kind: CronTab
|
|
metadata:
|
|
name: my-new-cron-object
|
|
spec:
|
|
cronSpec: "* * * * */5"
|
|
image: my-awesome-cron-image
|
|
```
|
|
|
|
and create it:
|
|
|
|
```shell
|
|
kubectl create -f my-crontab.yaml
|
|
```
|
|
|
|
You can specify the category using `kubectl get`:
|
|
|
|
```
|
|
kubectl get all
|
|
```
|
|
|
|
and it will include the custom resources of kind `CronTab`:
|
|
|
|
```console
|
|
NAME AGE
|
|
crontabs/my-new-cron-object 3s
|
|
```
|
|
|
|
{{% /capture %}}
|
|
|
|
{{% capture whatsnext %}}
|
|
* Learn how to [Migrate a ThirdPartyResource to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/).
|
|
* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions-k8s-io).
|
|
* Serve [multiple versions](custom-resource-definitions-versioning) of a
|
|
CustomResourceDefinition
|
|
{{% /capture %}}
|
|
|
|
|