---
layout: blog
title: "Enforce CRD Immutability with CEL Transition Rules"
date: 2022-09-29
slug: enforce-immutability-using-cel
---

**Author:** [Alexander Zielenski](https://github.com/alexzielenski) (Google)

Immutable fields can be found in a few places in the built-in Kubernetes types.
For example, you can't change the `.metadata.name` of an object. Specific objects
have fields where changes to existing objects are constrained; for example, the
`.spec.selector` of a Deployment.

Aside from simple immutability, there are other common design patterns such as
lists which are append-only, or a map with mutable values and immutable keys.

Until recently the best way to restrict field mutability for CustomResourceDefinitions
has been to create a validating
[admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#what-are-admission-webhooks):
this means a lot of complexity for the common case of making a field immutable.

Beta since Kubernetes 1.25, CEL Validation Rules allow CRD authors to express
validation constraints on their fields using a rich expression language,
[CEL](https://github.com/google/cel-spec). This article explores how you can
use validation rules to implement a few common immutability patterns directly in
the manifest for a CRD.

## Basics of validation rules

The new support for CEL validation rules in Kubernetes allows CRD authors to add
complicated admission logic for their resources without writing any code!

For example, A CEL rule to constrain a field `maximumSize` to be greater than a
`minimumSize` for a CRD might look like the following:

```yaml
rule: |
    self.maximumSize > self.minimumSize
message: 'Maximum size must be greater than minimum size.'
```

The rule field contains an expression written in CEL. `self` is a special keyword
in CEL which refers to the object whose type contains the rule.

The message field is an error message which will be sent to Kubernetes clients
whenever this particular rule is not satisfied.

For more details about the capabilities and limitations of Validation Rules using
CEL, please refer to
[validation rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules).
The [CEL specification](https://github.com/google/cel-spec) is also a good
reference for information specifically related to the language.

## Immutability patterns with CEL validation rules
This section implements several common use cases for immutability in Kubernetes
CustomResourceDefinitions, using validation rules expressed as
[kubebuilder marker comments](https://book.kubebuilder.io/reference/markers/crd.html).
Resultant OpenAPI generated by the kubebuilder marker comments will also be
included so that if you are writing your CRD manifests by hand you can still
follow along.

## Project setup

To use CEL rules with kubebuilder comments, you first need to set up a Golang
project structure with the CRD defined in Go.

You may skip this step if you are not using kubebuilder or are only interested
in the resultant OpenAPI extensions.

Begin with a folder structure of a Go module set up like the following. If
you have your own project already set up feel free to adapt this tutorial to your liking:

{{< mermaid >}}
graph LR
    . --> generate.go
    . --> pkg --> apis --> stable.example.com --> v1
    v1 --> doc.go
    v1 --> types.go
    . --> tools.go
{{</ mermaid >}}

This is the typical folder structure used by Kubernetes projects for defining new API resources.

`doc.go` contains package-level metadata such as the group and the version:
```go
// +groupName=stable.example.com
// +versionName=v1
package v1
```

`types.go` contains all type definitions in stable.example.com/v1

```go
package v1

import (
   metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// An empty CRD as an example of defining a type using controller tools
// +kubebuilder:storageversion
// +kubebuilder:subresource:status
type TestCRD struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   Spec   TestCRDSpec   `json:"spec,omitempty"`
   Status TestCRDStatus `json:"status,omitempty"`
}

type TestCRDStatus struct {}
type TestCRDSpec struct {
   // You will fill this in as you go along
}
```

`tools.go` contains a dependency on [controller-gen](https://book.kubebuilder.io/reference/generating-crd.html#generating-crds) which will be used to generate the CRD definition:

```go
//go:build tools

package celimmutabilitytutorial

// Force direct dependency on code-generator so that it may be executed with go run
import (
   _ "sigs.k8s.io/controller-tools/cmd/controller-gen"
)
```

Finally, `generate.go`contains a `go:generate` directive to make use of
`controller-gen`. `controller-gen` parses our `types.go` and creates generates
CRD yaml files into a `crd` folder:

```go
package celimmutabilitytutorial

//go:generate go run sigs.k8s.io/controller-tools/cmd/controller-gen crd paths=./pkg/apis/... output:dir=./crds
```


You may now want to add dependencies for our definitions and test the code generation:

```shell
cd cel-immutability-tutorial
go mod init <your-org>/<your-module-name>
go mod tidy
go generate ./...
```

After running these commands you now have completed the basic project structure.
Your folder tree should look like the following:

{{< mermaid >}}
graph LR
    . --> crds --> stable.example.com_testcrds.yaml
    . --> generate.go
    . --> go.mod
    . --> go.sum
    . --> pkg --> apis --> stable.example.com --> v1
    v1 --> doc.go
    v1 --> types.go
    . --> tools.go

{{</ mermaid >}}

The manifest for the example CRD is now available in `crds/stable.example.com_testcrds.yaml`.

## Immutablility after first modification
A common immutability design pattern is to make the field immutable once it has
been first set. This example will throw a validation error if the field after
changes after being first initialized.

```go
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.value) || has(self.value)", message="Value is required once set"
type ImmutableSinceFirstWrite struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   // +kubebuilder:validation:Optional
   // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
   // +kubebuilder:validation:MaxLength=512
   Value string `json:"value"`
}
```

The `+kubebuilder` directives in the comments inform controller-gen how to
annotate the generated OpenAPI. The `XValidation` rule causes the rule to appear
among the  `x-kubernetes-validations` OpenAPI extension. Kubernetes then
respects the OpenAPI spec to enforce our constraints.

To enforce a field's immutability after its first write, you need to apply the following constraints:
1. Field must be allowed to be initially unset `+kubebuilder:validation:Optional`
2. Once set, field must not be allowed to be removed: `!has(oldSelf.value) | has(self.value)` (type-scoped rule)
3. Once set, field must not be allowed to change value `self == oldSelf` (field-scoped rule)

Also note the additional directive `+kubebuilder:validation:MaxLength`. CEL
requires that all strings have attached max length so that it may estimate the
computation cost of the rule. Rules that are too expensive will be rejected.
For more information on CEL cost budgeting, check out the other tutorial.

### Example usage

Generating and installing the CRD should succeed:
```shell
# Ensure the CRD yaml is generated by controller-gen
go generate ./...
kubectl apply -f crds/stable.example.com_immutablesincefirstwrites.yaml
```
```console
customresourcedefinition.apiextensions.k8s.io/immutablesincefirstwrites.stable.example.com created
```

Creating initial empty object with no `value` is permitted since `value` is `optional`:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: ImmutableSinceFirstWrite
metadata:
  name: test1
EOF
```
```console
immutablesincefirstwrite.stable.example.com/test1 created
```

The initial modification of `value` succeeds:

```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: ImmutableSinceFirstWrite
metadata:
  name: test1
value: Hello, world!
EOF
```
```console
immutablesincefirstwrite.stable.example.com/test1 configured
```

An attempt to change `value` is blocked by the field-level validation rule. Note
the error message shown to the user comes from the validation rule.
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: ImmutableSinceFirstWrite
metadata:
  name: test1
value: Hello, new world!
EOF
```
```console
The ImmutableSinceFirstWrite "test1" is invalid: value: Invalid value: "string": Value is immutable
```

An attempt to remove the `value` field altogether is blocked by the other validation rule
on the type. The error message also comes from the rule.
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: ImmutableSinceFirstWrite
metadata:
  name: test1
EOF
```
```console
The ImmutableSinceFirstWrite "test1" is invalid: <nil>: Invalid value: "object": Value is required once set
```

### Generated schema
Note that in the generated schema there are two separate rule locations.
One is directly attached to the property `immutable_since_first_write`.
The other rule is associated with the crd type itself.

```yaml
openAPIV3Schema:
  properties:
    value:
      maxLength: 512
      type: string
      x-kubernetes-validations:
      - message: Value is immutable
        rule: self == oldSelf
  type: object
  x-kubernetes-validations:
  - message: Value is required once set
    rule: '!has(oldSelf.value) || has(self.value)'
```

## Immutability upon object creation

A field which is immutable upon creation time is implemented similarly to the
earlier example. The difference is that that field is marked required, and the
type-scoped rule is no longer necessary.

```go
type ImmutableSinceCreation struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   // +kubebuilder:validation:Required
   // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
   // +kubebuilder:validation:MaxLength=512
   Value string `json:"value"`
}
```

This field will be required when the object is created, and after that point will
not be allowed to be modified. Our CEL Validation Rule `self == oldSelf`

### Usage example

Generating and installing the CRD should succeed:
```shell
# Ensure the CRD yaml is generated by controller-gen
go generate ./...
kubectl apply -f crds/stable.example.com_immutablesincecreations.yaml
```
```console
customresourcedefinition.apiextensions.k8s.io/immutablesincecreations.stable.example.com created
```

Applying an object without the required field should fail:

```shell
kubectl apply -f - <<EOF
apiVersion: stable.example.com/v1
kind: ImmutableSinceCreation
metadata:
  name: test1
EOF
```
```console
The ImmutableSinceCreation "test1" is invalid:
* value: Required value
* <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation
```

Now that the field has been added, the operation is permitted:
```shell
kubectl apply -f - <<EOF
apiVersion: stable.example.com/v1
kind: ImmutableSinceCreation
metadata:
  name: test1
value: Hello, world!
EOF
```
```console
immutablesincecreation.stable.example.com/test1 created
```

If you attempt to change the `value`, the operation is blocked due to the
validation rules in the CRD. Note that the error message is as it was defined
in the validation rule.
```shell
kubectl apply -f - <<EOF
apiVersion: stable.example.com/v1
kind: ImmutableSinceCreation
metadata:
  name: test1
value: Hello, new world!
EOF
```
```console
The ImmutableSinceCreation "test1" is invalid: value: Invalid value: "string": Value is immutable
```

Also if you attempted to remove `value` altogether after adding it, you will
see an error as expected:
```shell
kubectl apply -f - <<EOF
apiVersion: stable.example.com/v1
kind: ImmutableSinceCreation
metadata:
  name: test1
EOF
```
```console
The ImmutableSinceCreation "test1" is invalid:
* value: Required value
* <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation
```

### Generated schema
```yaml
openAPIV3Schema:
  properties:
    value:
      maxLength: 512
      type: string
      x-kubernetes-validations:
      - message: Value is immutable
        rule: self == oldSelf
  required:
  - value
  type: object
```

## Append-only list of containers
In the case of ephemeral containers on Pods, Kubernetes enforces that the
elements in the list are immutable, and can’t be removed. The following example
shows how you could use CEL to achieve the same behavior.

```go
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.value) || has(self.value)", message="Value is required once set"
type AppendOnlyList struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   // +kubebuilder:validation:Optional
   // +kubebuilder:validation:MaxItems=100
   // +kubebuilder:validation:XValidation:rule="oldSelf.all(x, x in self)",message="Values may only be added"
   Values []v1.EphemeralContainer `json:"value"`
}
```

1. Once set, field must not be deleted: `!has(oldSelf.value) || has(self.value)` (type-scoped)
2. Once a value is added it is not removed: `oldSelf.all(x, x in self)` (field-scoped)
2. Value may be initially unset: `+kubebuilder:validation:Optional`

Note that for cost-budgeting purposes, `MaxItems` is also required to be specified.

### Example usage

Generating and installing the CRD should succeed:
```shell
# Ensure the CRD yaml is generated by controller-gen
go generate ./...
kubectl apply -f crds/stable.example.com_appendonlylists.yaml
```
```console
customresourcedefinition.apiextensions.k8s.io/appendonlylists.stable.example.com created
```

Creating an initial list with one element inside should succeed without problem:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: AppendOnlyList
metadata:
  name: testlist
value:
  - name: container1
    image: nginx/nginx
EOF
```
```console
appendonlylist.stable.example.com/testlist created
```

Adding an element to the list should also proceed without issue as expected:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: AppendOnlyList
metadata:
  name: testlist
value:
  - name: container1
    image: nginx/nginx
  - name: container2
    image: mongodb/mongodb
EOF
```
```console
appendonlylist.stable.example.com/testlist configured
```

But if you now attempt to remove an element, the error from the validation rule
is triggered:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: AppendOnlyList
metadata:
  name: testlist
value:
  - name: container1
    image: nginx/nginx
EOF
```
```console
The AppendOnlyList "testlist" is invalid: value: Invalid value: "array": Values may only be added
```

Additionally, to attempt to remove the field once it has been set is also disallowed
by the type-scoped validation rule.
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: AppendOnlyList
metadata:
  name: testlist
EOF
```
```console
The AppendOnlyList "testlist" is invalid: <nil>: Invalid value: "object": Value is required once set
```

### Generated schema
```yaml
openAPIV3Schema:
  properties:
    value:
      items: ...
      maxItems: 100
      type: array
      x-kubernetes-validations:
      - message: Values may only be added
        rule: oldSelf.all(x, x in self)
  type: object
  x-kubernetes-validations:
  - message: Value is required once set
    rule: '!has(oldSelf.value) || has(self.value)'
```

## Map with append-only keys, immutable values

```go
// A map which does not allow keys to be removed or their values changed once set. New keys may be added, however.
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.values) || has(self.values)", message="Value is required once set"
type MapAppendOnlyKeys struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	// +kubebuilder:validation:Optional
	// +kubebuilder:validation:MaxProperties=10
	// +kubebuilder:validation:XValidation:rule="oldSelf.all(key, key in self && self[key] == oldSelf[key])",message="Keys may not be removed and their values must stay the same"
	Values map[string]string `json:"values,omitempty"`
}
```

1. Once set, field must not be deleted: `!has(oldSelf.values) || has(self.values)` (type-scoped)
2. Once a key is added it is not removed nor is its value modified: `oldSelf.all(key, key in self && self[key] == oldSelf[key])` (field-scoped)
3. Value may be initially unset: `+kubebuilder:validation:Optional`

### Example usage
Generating and installing the CRD should succeed:
```shell
# Ensure the CRD yaml is generated by controller-gen
go generate ./...
kubectl apply -f crds/stable.example.com_mapappendonlykeys.yaml
```
```console
customresourcedefinition.apiextensions.k8s.io/mapappendonlykeys.stable.example.com created
```

Creating an initial object with one key within `values` should be permitted:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: MapAppendOnlyKeys
metadata:
  name: testmap
values:
    key1: value1
EOF
```
```console
mapappendonlykeys.stable.example.com/testmap created
```

Adding new keys to the map should also be permitted:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: MapAppendOnlyKeys
metadata:
  name: testmap
values:
    key1: value1
    key2: value2
EOF
```
```console
mapappendonlykeys.stable.example.com/testmap configured
```

But if a key is removed, the error messagr from the validation rule should be
returned:
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: MapAppendOnlyKeys
metadata:
  name: testmap
values:
    key1: value1
EOF
```
```console
The MapAppendOnlyKeys "testmap" is invalid: values: Invalid value: "object": Keys may not be removed and their values must stay the same
```

If the entire field is removed, the other validation rule is triggered and the
operation is prevented. Note that the error message for the validation rule is
shown to the user.
```shell
kubectl apply -f - <<EOF
---
apiVersion: stable.example.com/v1
kind: MapAppendOnlyKeys
metadata:
  name: testmap
EOF
```
```console
The MapAppendOnlyKeys "testmap" is invalid: <nil>: Invalid value: "object": Value is required once set
```

### Generated schema

```yaml
openAPIV3Schema:
  description: A map which does not allow keys to be removed or their values
    changed once set. New keys may be added, however.
  properties:
    values:
      additionalProperties:
        type: string
      maxProperties: 10
      type: object
      x-kubernetes-validations:
      - message: Keys may not be removed and their values must stay the same
        rule: oldSelf.all(key, key in self && self[key] == oldSelf[key])
  type: object
  x-kubernetes-validations:
  - message: Value is required once set
    rule: '!has(oldSelf.values) || has(self.values)'
```

# Going further
The above examples showed how CEL rules can be added to kubebuilder types.
The same rules can be added directly to OpenAPI if writing a manifest for a CRD by hand.

For native types, the same behavior can be achieved using kube-openapi’s marker
[`+validations`](https://github.com/kubernetes/kube-openapi/blob/923526ac052c59656d41710b45bbcb03748aa9d6/pkg/generators/extension.go#L69).

Usage of CEL within Kubernetes Validation Rules is so much more powerful than
what has been shown in this article. For more information please check out
[validation rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules)
in the Kubernetes documentation and [CRD Validation Rules Beta](https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/) blog post.