From 89457360295354aadeaf6b6948b8ecb99482c89b Mon Sep 17 00:00:00 2001 From: Craig Box Date: Wed, 29 Mar 2023 22:03:04 +1300 Subject: [PATCH] address nits and rename file for new date --- ...ape-validating-admission-policy-library.md | 235 ++++++++++++++++++ ...ape-validating-admission-policy-library.md | 158 ------------ 2 files changed, 235 insertions(+), 158 deletions(-) create mode 100644 content/en/blog/_posts/2023-03-30-kubescape-validating-admission-policy-library.md delete mode 100644 content/en/blog/_posts/2023-04-05-kubescape-validating-admission-policy-library.md diff --git a/content/en/blog/_posts/2023-03-30-kubescape-validating-admission-policy-library.md b/content/en/blog/_posts/2023-03-30-kubescape-validating-admission-policy-library.md new file mode 100644 index 00000000000..0ba763d8b24 --- /dev/null +++ b/content/en/blog/_posts/2023-03-30-kubescape-validating-admission-policy-library.md @@ -0,0 +1,235 @@ +--- +layout: blog +title: "Kubernetes Validating Admission Policies: A Practical Example" +date: 2023-03-30T00:00:00+0000 +slug: kubescape-validating-admission-policy-library +--- + +**Authors**: Craig Box (ARMO), Ben Hirschberg (ARMO) + +Admission control is an important part of the Kubernetes control plane, with several internal +features depending on the ability to approve or change an API object as it is submitted to the +server. It is also useful for an administrator to be able to define business logic, or policies, +regarding what objects can be admitted into a cluster. To better support that use case, [Kubernetes +introduced external admission control in +v1.7](https://kubernetes.io/blog/2017/06/kubernetes-1-7-security-hardening-stateful-application-extensibility-updates/). + +In addition to countless custom, internal implementations, many open source projects and commercial +solutions implement admission controllers with user-specified policy, including +[Kyverno](https://github.com/kyverno/kyverno) and Open Policy Agent’s +[Gatekeeper](https://github.com/open-policy-agent/gatekeeper). + +While admission controllers for policy have seen adoption, there are blockers for their widespread +use. Webhook infrastructure must be maintained as a production service, with all that entails. The +failure case of an admission control webhook must either be closed, reducing the availability of the +cluster; or open, negating the use of the feature for policy enforcement. The network hop and +evaluation time makes admission control a notable component of latency when dealing with, for +example, pods being spun up to respond to a network request in a "serverless" environment. + +## Validating admission policies and the Common Expression Language + +Version 1.26 of Kubernetes introduced, in alpha, a compromise solution. [Validating admission +policies](/docs/reference/access-authn-authz/validating-admission-policy/) are a declarative, +in-process alternative to admission webhooks. They use the [Common Expression +Language](https://github.com/google/cel-spec) (CEL) to declare validation rules. + +CEL was developed by Google for security and policy use cases, based on learnings from the Firebase +real-time database. Its design allows it to be safely embedded into applications and executed in +microseconds, with limited compute and memory impact. [Validation rules for +CRDs](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules) +introduced CEL to the Kubernetes ecosystem in v1.23, and at the time it was noted that the language +would suit a more generic implementation of validation by admission control. + +## Giving CEL a roll - a practical example + +[Kubescape](https://github.com/kubescape/kubescape) is a CNCF project which has become one of the +most popular ways for users to improve the security posture of a Kubernetes cluster and validate its +compliance. Its [controls](https://github.com/kubescape/regolibrary) — groups of tests against API +objects — are built in [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/), the +policy language of Open Policy Agent. + +Rego has a reputation for complexity, based largely on the fact that it is a declarative query +language (like SQL). It [was +considered](https://github.com/kubernetes/enhancements/blob/499e28/keps/sig-api-machinery/2876-crd-validation-expression-language/README.md#alternatives) +for use in Kubernetes, but it does not offer the same sandbox constraints as CEL. + +A common feature request for the project is to be able to implement policies based on Kubescape’s +findings and output. For example, after scanning pods for [known paths to cloud credential +files](https://hub.armosec.io/docs/c-0020), users would like the ability to enforce policy that +these pods should not be admitted at all. The Kubescape team thought this would be the perfect +opportunity to try and port our existing controls to CEL and apply them as admission policies. + +### Show me the policy + +It did not take us long to convert many of our controls and build a [library of validating admission +policies](https://github.com/kubescape/cel-admission-library). Let’s look at one as an example. + +Kubescape’s [control C-0017](https://hub.armosec.io/docs/c-0017) covers the requirement for +containers to have an immutable (read-only) root filesystem. This is a best practice according to +the [NSA Kubernetes hardening +guidelines](/blog/2021/10/05/nsa-cisa-kubernetes-hardening-guidance/#immutable-container-filesystems), +but is not currently required as a part of any of the [pod security +standards](/docs/concepts/security/pod-security-standards/). + +Here's how we implemented it in CEL: + +```yaml +apiVersion: admissionregistration.k8s.io/v1alpha1 +kind: ValidatingAdmissionPolicy +metadata: + name: "kubescape-c-0017-deny-resources-with-mutable-container-filesystem" +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["pods"] + - apiGroups: ["apps"] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["deployments","replicasets","daemonsets","statefulsets"] + - apiGroups: ["batch"] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["jobs","cronjobs"] + validations: + - expression: "object.kind != 'Pod' || object.spec.containers.all(container, has(container.securityContext) && has(container.securityContext.readOnlyRootFilesystem) && container.securityContext.readOnlyRootFilesystem == true)" + message: "Pods having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017)" + - expression: "['Deployment','ReplicaSet','DaemonSet','StatefulSet','Job'].all(kind, object.kind != kind) || object.spec.template.spec.containers.all(container, has(container.securityContext) && has(container.securityContext.readOnlyRootFilesystem) && container.securityContext.readOnlyRootFilesystem == true)" + message: "Workloads having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017)" + - expression: "object.kind != 'CronJob' || object.spec.jobTemplate.spec.template.spec.containers.all(container, has(container.securityContext) && has(container.securityContext.readOnlyRootFilesystem) && container.securityContext.readOnlyRootFilesystem == true)" + message: "CronJob having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017)" +``` + +Match constraints are provided for three possible API groups: the `core/v1` group for Pods, the +`apps/v1` workload controllers, and the `batch/v1` job controllers. + +{{< note >}} `matchConstraints` will convert the API object to the matched version for you. If, for +example, an API request was for `apps/v1beta1` and you match `apps/v1` in matchConstraints, the API +request will be converted from `apps/v1beta1` to `apps/v1` and then validated. This has the useful +property of making validation rules secure against the introduction of new versions of APIs, which +would otherwise allow API requests to sneak past the validation rule by using the newly introduced +version. {{< /note >}} + +The `validations` include the CEL rules for the objects. There are three different expressions, +catering for the fact that a Pod `spec` can be at the root of the object (a [naked +pod](https://kubernetes.io/docs/concepts/configuration/overview/#naked-pods-vs-replicasets-deployments-and-jobs)), +under `template` (a workload controller or a Job), or under `jobTemplate` (a CronJob). + +In the event that any `spec` does not have `readOnlyRootFilesystem` set to true, the object will not +be admitted. + +{{< note >}} In our initial release, we have grouped the three expressions into the same policy +object. This means they can be enabled and disabled atomically, and thus there is no chance that a +user will accidentally leave a compliance gap by enabling policy for one API group and not the +others. Breaking them into separate policies would allow us access to improvements targeted for the +1.27 release, including type checking. We are talking to SIG API Machinery about how to best address +this before the APIs reach `v1`. {{< /note >}} + +### Using the CEL library in your cluster + +Policies are provided as Kubernetes objects, which are then bound to certain resources by a +[selector](/docs/concepts/overview/working-with-objects/labels/#label-selectors). + +[Minikube](https://minikube.sigs.k8s.io/docs/) is a quick and easy way to install and configure a +Kubernetes cluster for testing. To install Kubernetes v1.26 with the `ValidatingAdmissionPolicy` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled: + +```shell +minikube start --kubernetes-version=1.26.1 --extra-config=apiserver.runtime-config=admissionregistration.k8s.io/v1alpha1 --feature-gates='ValidatingAdmissionPolicy=true' +``` + +To install the policies in your cluster: + +```shell +# Install configuration CRD +kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/policy-configuration-definition.yaml +# Install basic configuration +kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/basic-control-configuration.yaml +# Install policies +kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/kubescape-validating-admission-policies.yaml +``` + +To apply policies to objects, create a `ValidatingAdmissionPolicyBinding` resource. Let’s apply the +above Kubescape C-0017 control to any namespace with the label `policy=enforced`: + +```shell +# Create a binding +kubectl apply -f - <}} -`matchConstraints` will convert the API object to the matched version for you. If, for example, an API request was for `apps/v1beta1` and you match `apps/v1` in matchConstraints, the API request will be converted from `apps/v1beta1` to `apps/v1` and then validated. This has the useful property of making validation rules secure against the introduction of new versions of APIs, which would otherwise allow API requests to sneak past the validation rule by using the newly introduced version. -{{< /note >}} - -The `validations` include the CEL rules for the objects. There are three different expressions, catering for the fact that a Pod `spec` can be at the root of the object (a [naked pod](https://kubernetes.io/docs/concepts/configuration/overview/#naked-pods-vs-replicasets-deployments-and-jobs)), under `template` (a workload controller or a Job), or under `jobTemplate` (a CronJob). - -In the event that any `spec` does not have `readOnlyRootFilesystem` set to true, the object will not be admitted. - -{{< note >}} -In our initial release, we have grouped the three expressions into the same policy object. This means they can be enabled and disabled atomically, and thus there is no chance that a user will accidentally leave a compliance gap by enabling policy for one API group and not the others. Breaking them into separate policies would allow us access to improvements targeted for the 1.27 release, including type checking. We are talking to SIG API Machinery about how to best address this before the APIs reach `v1`. -{{< /note >}} - -### Using the CEL library in your cluster - -Policies are provided as Kubernetes objects, which are then bound to certain resources by a [selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). - -[Minikube](https://minikube.sigs.k8s.io/docs/) is a quick and easy way to install and configure a Kubernetes cluster for testing. To install Kubernetes v1.26 with the `ValidatingAdmissionPolicy` [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) enabled: - -```shell -minikube start --kubernetes-version=1.26.1 --extra-config=apiserver.runtime-config=admissionregistration.k8s.io/v1alpha1 --feature-gates='ValidatingAdmissionPolicy=true' -``` - -To install the policies in your cluster: - -```shell -# Install configuration CRD -kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/policy-configuration-definition.yaml -# Install basic configuration -kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/basic-control-configuration.yaml -# Install policies -kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/kubescape-validating-admission-policies.yaml -``` - -To apply policies to objects, create a `ValidatingAdmissionPolicyBinding` resource. Let’s apply the above Kubescape C-0017 control to any namespace with the label `policy=enforced`: - -```shell -# Create a binding -kubectl apply -f - <