Volume Populators Redesign Blog
For https://github.com/kubernetes/enhancements/issues/1495pull/28970/head
Ben Swartzlander
parent
575bd04640
commit
c0b5d85371
|
|
@ -0,0 +1,219 @@
|
|||
---
|
||||
layout: blog
|
||||
title: "Kubernetes 1.22: A New Design for Volume Populators"
|
||||
date: 2021-08-30
|
||||
slug: volume-populators-redesigned
|
||||
---
|
||||
|
||||
**Authors:**
|
||||
Ben Swartzlander (NetApp)
|
||||
|
||||
Kubernetes v1.22, released earlier this month, introduced a redesigned approach for volume
|
||||
populators. Originally implemented
|
||||
in v1.18, the API suffered from backwards compatibility issues. Kubernetes v1.22 includes a new API
|
||||
field called `dataSourceRef` that fixes these problems.
|
||||
|
||||
## Data sources
|
||||
|
||||
Earlier Kubernetes releases already added a `dataSource` field into the
|
||||
[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) API,
|
||||
used for cloning volumes and creating volumes from snapshots. You could use the `dataSource` field when
|
||||
creating a new PVC, referencing either an existing PVC or a VolumeSnapshot in the same namespace.
|
||||
That also modified the normal provisioning process so that instead of yielding an empty volume, the
|
||||
new PVC contained the same data as either the cloned PVC or the cloned VolumeSnapshot.
|
||||
|
||||
Volume populators embrace the same design idea, but extend it to any type of object, as long
|
||||
as there exists a [custom resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
|
||||
to define the data source, and a populator controller to implement the logic. Initially,
|
||||
the `dataSource` field was directly extended to allow arbitrary objects, if the `AnyVolumeDataSource`
|
||||
feature gate was enabled on a cluster. That change unfortunately caused backwards compatibility
|
||||
problems, and so the new `dataSourceRef` field was born.
|
||||
|
||||
In v1.22 if the `AnyVolumeDataSource` feature gate is enabled, the `dataSourceRef` field is
|
||||
added, which behaves similarly to the `dataSource` field except that it allows arbitrary
|
||||
objects to be specified. The API server ensures that the two fields always have the same
|
||||
contents, and neither of them are mutable. The differences is that at creation time
|
||||
`dataSource` allows only PVCs or VolumeSnapshots, and ignores all other values, while
|
||||
`dataSourceRef` allows most types of objects, and in the few cases it doesn't allow an
|
||||
object (core objects other than PVCs) a validation error occurs.
|
||||
|
||||
When this API change graduates to stable, we would deprecate using `dataSource` and recommend
|
||||
using `dataSourceRef` field for all use cases.
|
||||
In the v1.22 release, `dataSourceRef` is available (as an alpha feature) specifically for cases
|
||||
where you want to use for custom volume populators.
|
||||
|
||||
## Using populators
|
||||
|
||||
Every volume populator must have one or more CRDs that it supports. Administrators may
|
||||
install the CRD and the populator controller and then PVCs with a `dataSourceRef` specifies
|
||||
a CR of the type that the populator supports will be handled by the populator controller
|
||||
instead of the CSI driver directly.
|
||||
|
||||
Underneath the covers, the CSI driver is still invoked to create an empty volume, which
|
||||
the populator controller fills with the appropriate data. The PVC doesn't bind to the PV
|
||||
until it's fully populated, so it's safe to define a whole application manifest including
|
||||
pod and PVC specs and the pods won't begin running until everything is ready, just as if
|
||||
the PVC was a clone of another PVC or VolumeSnapshot.
|
||||
|
||||
## How it works
|
||||
|
||||
PVCs with data sources are still noticed by the external-provisioner sidecar for the
|
||||
related storage class (assuming a CSI provisioner is used), but because the sidecar
|
||||
doesn't understand the data source kind, it doesn't do anything. The populator controller
|
||||
is also watching for PVCs with data sources of a kind that it understands and when it
|
||||
sees one, it creates a temporary PVC of the same size, volume mode, storage class,
|
||||
and even on the same topology (if topology is used) as the original PVC. The populator
|
||||
controller creates a worker pod that attaches to the volume and writes the necessary
|
||||
data to it, then detaches from the volume and the populator controller rebinds the PV
|
||||
from the temporary PVC to the orignal PVC.
|
||||
|
||||
## Trying it out
|
||||
|
||||
The following things are required to use volume populators:
|
||||
* Enable the `AnyVolumeDataSource` feature gate
|
||||
* Install a CRD for the specific data source / populator
|
||||
* Install the populator controller itself
|
||||
|
||||
Populator controllers may use the [lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator)
|
||||
library to do most of the Kubernetes API level work. Individual populators only need to
|
||||
provide logic for actually writing data into the volume based on a particular CR
|
||||
instance. This library provides a sample populator implementation.
|
||||
|
||||
These optional components improve user experience:
|
||||
* Install the VolumePopulator CRD
|
||||
* Create a VolumePopulator custom respource for each specific data source
|
||||
* Install the [volume data source validator](https://github.com/kubernetes-csi/volume-data-source-validator)
|
||||
controller (alpha)
|
||||
|
||||
The purpose of these components is to generate warning events on PVCs with data sources
|
||||
for which there is no populator.
|
||||
|
||||
## Putting it all together
|
||||
|
||||
To see how this works, you can install the sample "hello" populator and try it
|
||||
out.
|
||||
|
||||
First install the volume-data-source-validator controller.
|
||||
|
||||
```terminal
|
||||
kubectl apply -f https://github.com/kubernetes-csi/volume-data-source-validator/blob/master/deploy/kubernetes/rbac-data-source-validator.yaml
|
||||
kubectl apply -f https://github.com/kubernetes-csi/volume-data-source-validator/blob/master/deploy/kubernetes/setup-data-source-validator.yaml
|
||||
```
|
||||
|
||||
Next install the example populator.
|
||||
|
||||
```terminal
|
||||
kubectl apply -f https://github.com/kubernetes-csi/lib-volume-populator/blob/master/example/hello-populator/crd.yaml
|
||||
kubectl apply -f https://github.com/kubernetes-csi/lib-volume-populator/blob/master/example/hello-populator/deploy.yaml
|
||||
```
|
||||
|
||||
Create an instance of the `Hello` CR, with some text.
|
||||
|
||||
```yaml
|
||||
apiVersion: hello.k8s.io/v1alpha1
|
||||
kind: Hello
|
||||
metadata:
|
||||
name: example-hello
|
||||
spec:
|
||||
fileName: example.txt
|
||||
fileContents: Hello, world!
|
||||
```
|
||||
|
||||
Create a PVC that refers to that CR as its data source.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: example-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Mi
|
||||
dataSourceRef:
|
||||
apiGroup: hello.k8s.io
|
||||
kind: Hello
|
||||
name: example-hello
|
||||
volumeMode: Filesystem
|
||||
```
|
||||
|
||||
Next, run a job that reads the file in the PVC.
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: example-job
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: example-container
|
||||
image: busybox:latest
|
||||
command:
|
||||
- cat
|
||||
- /mnt/example.txt
|
||||
volumeMounts:
|
||||
- name: vol
|
||||
mountPath: /mnt
|
||||
restartPolicy: Never
|
||||
volumes:
|
||||
- name: vol
|
||||
persistentVolumeClaim:
|
||||
claimName: example-pvc
|
||||
```
|
||||
|
||||
Wait for the job to complete (including all of its dependencies).
|
||||
|
||||
```terminal
|
||||
kubectl wait --for=condition=Complete job/example-job
|
||||
```
|
||||
|
||||
And last examine the log from the job.
|
||||
|
||||
```terminal
|
||||
kubectl logs job/example-job
|
||||
Hello, world!
|
||||
```
|
||||
|
||||
Note that the volume already contained a text file with the string contents from
|
||||
the CR. This is only the simplest example. Actual populators can set up the volume
|
||||
to contain arbitrary contents.
|
||||
|
||||
## How to write your own volume populator
|
||||
|
||||
Developers interested in writing new poplators are encouraged to use the
|
||||
[lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library
|
||||
and to only supply a small controller wrapper around the library, and a pod image
|
||||
capable of attaching to volumes and writing the appropriate data to the volume.
|
||||
|
||||
Individual populators can be extremely generic such that they work with every type
|
||||
of PVC, or they can do vendor specific things to rapidly fill a volume with data
|
||||
if the volume was provisioned by a specific CSI driver from the same vendor, for
|
||||
example, by communicating directly with the storage for that volume.
|
||||
|
||||
## The future
|
||||
|
||||
As this feature is still in alpha, we expect to update the out of tree controllers
|
||||
with more tests and documentation. The community plans to eventually re-implement
|
||||
the populator library as a sidecar, for ease of operations.
|
||||
|
||||
We hope to see some official community-supported populators for some widely-shared
|
||||
use cases. Also, we expect that volume populators will be used by backup vendors
|
||||
as a way to "restore" backups to volumes, and possibly a standardized API to do
|
||||
this will evolve.
|
||||
|
||||
## How can I learn more?
|
||||
|
||||
The enhancement proposal,
|
||||
[Volume Populators](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators), includes lots of detail about the history and technical implementation
|
||||
of this feature.
|
||||
|
||||
[Volume populators and data sources] (in the documenation topic about persistent volumes)
|
||||
explains how to use this feature in your cluster.
|
||||
|
||||
Please get involved by joining the Kubernetes storage SIG to help us enhance this
|
||||
feature. There are a lot of good ideas already and we'd be thrilled to have more!
|
||||
|
||||
Loading…
Reference in New Issue