From 48fdee2334774b22c53135f22afd9e674b16cb70 Mon Sep 17 00:00:00 2001 From: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com> Date: Tue, 12 Mar 2024 23:01:14 +0000 Subject: [PATCH 1/2] docs: adds documentation for Storage Version Migration Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com> --- .../storage-version-migration.md | 14 + .../storage-version-migration.md | 312 ++++++++++++++++++ 2 files changed, 326 insertions(+) create mode 100644 content/en/docs/reference/command-line-tools-reference/feature-gates/storage-version-migration.md create mode 100644 content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/storage-version-migration.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/storage-version-migration.md new file mode 100644 index 0000000000..a1f8187794 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/storage-version-migration.md @@ -0,0 +1,14 @@ +--- +title: StorageVersionMigration +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.30" + toVersion: "1.32" +--- +Enables storage version migration. See [Migrate Kubernetes Objects Using Storage Version Migration](/docs/tasks/manage-kubernetes-objects/storage-version-migration) for more details. diff --git a/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md b/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md new file mode 100644 index 0000000000..fbe59852fe --- /dev/null +++ b/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md @@ -0,0 +1,312 @@ +--- +title: Migrate Kubernetes Objects Using Storage Version Migration + +reviewers: + - deads2k + - jpbetz + - enj + - nilekhc + +content_type: task +min-kubernetes-server-version: v1.30 +weight: 60 +--- + + +{{< feature-state feature_gate_name="StorageVersionMigration" >}} + +Kubernetes relies on API data being actively re-written, to support some +maintenance activities related to at rest storage. Two prominent examples are +the versioned schema of stored resources (that is, the preferred storage schema +changing from v1 to v2 for a given resource) and encryption at rest +(that is, rewriting stale data based on a change in how the data should be encrypted). + +## {{% heading "prerequisites" %}} + +Install [`kubectl`](/docs/tasks/tools/#kubectl). + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + +## Re-encrypt Kubernetes secrets using storage version migration +- To begin with, [configure KMS provider](/docs/tasks/administer-cluster/kms-provider/) +to encrypt data at rest in etcd using following encryption configuration. + ```yaml + kind: EncryptionConfiguration + apiVersion: apiserver.config.k8s.io/v1 + resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: c2VjcmV0IGlzIHNlY3VyZQ== + ``` + Make sure to enable automatic reload of encryption +configuration file by setting `--encryption-provider-config-automatic-reload` to true. +- Create a Secret using kubectl. + ```shell + kubectl create secret generic my-secret --from-literal=key1=supersecre + ``` +- [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted) + the serialized data for that Secret object is prefixed with `k8s:enc:aescbc:v1:key1`. +- Update the encryption configuration file as follows to rotate the encryption key. + ```yaml + kind: EncryptionConfiguration + apiVersion: apiserver.config.k8s.io/v1 + resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key2 + secret: c2VjcmV0IGlzIHNlY3VyZSwgaXMgaXQ/ + - aescbc: + keys: + - name: key1 + secret: c2VjcmV0IGlzIHNlY3VyZQ== + ``` +- To ensure that previously created secret `my-secert` is re-encrypted +with new key `key2`, you will use _Storage Version Migration_. +- Create `StorageVersionMigration` resource file with the name `migrate-secret.yaml` +as follows: + ```yaml + kind: StorageVersionMigration + apiVersion: storagemigration.k8s.io/v1alpha1 + metadata: + name: secrets-migration + spec: + resource: + group: "" + version: v1 + resource: secrets + ``` + Create the resource using _kubectl_ as follows: + ```shell + kubectl apply -f migrate-secret.yaml + ``` +- Monitor migration of Secrets using status. Successful migration should have +_Succeeded_ condition set to _true_ in the status field. Get the migration resource +as follows: + ```shell + kubectl get storageversionmigration.storagemigration.k8s.io/secrets-migration -o yaml + ``` + This will produce an output similar to: + ```yaml + kind: StorageVersionMigration + apiVersion: storagemigration.k8s.io/v1alpha1 + metadata: + name: secrets-migration + uid: 628f6922-a9cb-4514-b076-12d3c178967c + resourceVersion: '90' + creationTimestamp: '2024-03-12T20:29:45Z' + spec: + resource: + group: "" + version: v1 + resource: secrets + status: + conditions: + - type: Running + status: 'False' + lastUpdateTime: '2024-03-12T20:29:46Z' + reason: StorageVersionMigrationInProgress + - type: Succeeded + status: 'True' + lastUpdateTime: '2024-03-12T20:29:46Z' + reason: StorageVersionMigrationSucceeded + resourceVersion: '84' + ``` +- [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted) + the stored secret is now prefixed with `k8s:enc:aescbc:v1:key2`. + +## Update the preferred storage schema of the resource while moving from _v1_ to _v2_ +Consider a scenario where a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} +(CRD) is created to serve custom resources (CRs) and is set as the preferred storage schema. When it's time +to introduce v2 of the CRD, it can be added for serving only with a conversion +webhook. This enables a smoother transition where users can create CRs using +either the v1 or v2 schema, with the webhook in place to perform the necessary +schema conversion between them. Before setting v2 as the preferred storage schema +version, it's important to ensure that all existing CRs stored as v1 are migrated to v2. +This migration can be achieved through _Storage Version Migration_ to migrate all CRs from v1 to v2. + +- Create a manifest for the CRD, named `test-crd.yaml`, as follows: + ```yaml + apiVersion: apiextensions.k8s.io/v1 + kind: CustomResourceDefinition + metadata: + name: selfierequests.stable.example.com + spec: + group: stable.example.com + names: + plural: SelfieRequests + singular: SelfieRequest + kind: SelfieRequest + listKind: SelfieRequestList + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + hostPort: + type: string + conversion: + strategy: Webhook + webhook: + clientConfig: + url: https://127.0.0.1:9443/crdconvert + caBundle: + conversionReviewVersions: + - v1 + - v2 + ``` + Create CRD using kubectl + ```shell + kubectl apply -f test-crd.yaml + ``` +- Create a manifest for an example testcrd. Name the manifest `cr1.yaml` and use these contents: + ```yaml + apiVersion: stable.example.com/v1 + kind: SelfieRequest + metadata: + name: cr1 + namespace: default + ``` + Create CR using kubectl + ```shell + kubectl apply -f cr1.yaml + ``` +- Verify that CR is written and stored as v1 by getting the object from etcd. + ```shell + ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C + ``` + where [...] contains the additional arguments for connecting to the etcd server. +- Update the CRD `test-crd.yaml` to include v2 version for serving and storage + and v1 as serving only, as follows: + ```yaml + apiVersion: apiextensions.k8s.io/v1 + kind: CustomResourceDefinition + metadata: + name: selfierequests.stable.example.com + spec: + group: stable.example.com + names: + plural: SelfieRequests + singular: SelfieRequest + kind: SelfieRequest + listKind: SelfieRequestList + scope: Namespaced + versions: + - name: v2 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + host: + type: string + port: + type: string + - name: v1 + served: true + storage: false + schema: + openAPIV3Schema: + type: object + properties: + hostPort: + type: string + conversion: + strategy: Webhook + webhook: + clientConfig: + url: 'https://127.0.0.1:9443/crdconvert' + caBundle: + conversionReviewVersions: + - v1 + - v2 + ``` + Update CRD using kubectl + ```shell + kubectl apply -f test-crd.yaml + ``` +- Create CR resource file with name `cr2.yaml` as follows: + ```yaml + apiVersion: stable.example.com/v2 + kind: SelfieRequest + metadata: + name: cr2 + namespace: default + ``` +- Create CR using kubectl + ```shell + kubectl apply -f cr2.yaml + ``` +- Verify that CR is written and stored as v2 by getting the object from etcd. + ```shell + ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr2 [...] | hexdump -C + ``` + where [...] contains the additional arguments for connecting to the etcd server. +- Create `StorageVersionMigration` resource file with the name `migrate-crd.yaml` +as follows: + ```yaml + kind: StorageVersionMigration + apiVersion: storagemigration.k8s.io/v1alpha1 + metadata: + name: crdsvm + spec: + resource: + group: stable.example.com + version: v1 + resource: SelfieRequest + ``` + Create the resource using _kubectl_ as follows: + ```shell + kubectl apply -f migrate-crd.yaml + ``` +- Monitor migration of secrets using status. Successful migration should have + _Succeeded_ condition set to _true_ in the status field. Get the migration resource + as follows: + ```shell + kubectl get storageversionmigration.storagemigration.k8s.io/crdsvm -o yaml + ``` + This will produce an output similar to: + ```yaml + kind: StorageVersionMigration + apiVersion: storagemigration.k8s.io/v1alpha1 + metadata: + name: crdsvm + uid: 13062fe4-32d7-47cc-9528-5067fa0c6ac8 + resourceVersion: '111' + creationTimestamp: '2024-03-12T22:40:01Z' + spec: + resource: + group: stable.example.com + version: v1 + resource: testcrds + status: + conditions: + - type: Running + status: 'False' + lastUpdateTime: '2024-03-12T22:40:03Z' + reason: StorageVersionMigrationInProgress + - type: Succeeded + status: 'True' + lastUpdateTime: '2024-03-12T22:40:03Z' + reason: StorageVersionMigrationSucceeded + resourceVersion: '106' + ``` +- Verify that previously created cr1 is now written and stored as v2 by getting the object from etcd. + ```shell + ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C + ``` + where [...] contains the additional arguments for connecting to the etcd server. From 048f53da0225f32f6003747139fef2412e692c52 Mon Sep 17 00:00:00 2001 From: drewhagen Date: Wed, 27 Mar 2024 21:47:28 -0500 Subject: [PATCH 2/2] docs: PR feedback for documentation on Storage Version Migration --- .../storage-version-migration.md | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md b/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md index fbe59852fe..a9b3378b26 100644 --- a/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md +++ b/content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md @@ -39,7 +39,7 @@ to encrypt data at rest in etcd using following encryption configuration. resources: - resources: - secrets - providers: + providers: - aescbc: keys: - name: key1 @@ -49,7 +49,7 @@ to encrypt data at rest in etcd using following encryption configuration. configuration file by setting `--encryption-provider-config-automatic-reload` to true. - Create a Secret using kubectl. ```shell - kubectl create secret generic my-secret --from-literal=key1=supersecre + kubectl create secret generic my-secret --from-literal=key1=supersecret ``` - [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted) the serialized data for that Secret object is prefixed with `k8s:enc:aescbc:v1:key1`. @@ -60,7 +60,7 @@ configuration file by setting `--encryption-provider-config-automatic-reload` to resources: - resources: - secrets - providers: + providers: - aescbc: keys: - name: key2 @@ -72,8 +72,7 @@ configuration file by setting `--encryption-provider-config-automatic-reload` to ``` - To ensure that previously created secret `my-secert` is re-encrypted with new key `key2`, you will use _Storage Version Migration_. -- Create `StorageVersionMigration` resource file with the name `migrate-secret.yaml` -as follows: +- Create a StorageVersionMigration manifest named `migrate-secret.yaml` as follows: ```yaml kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 @@ -85,17 +84,19 @@ as follows: version: v1 resource: secrets ``` - Create the resource using _kubectl_ as follows: + Create the object using _kubectl_ as follows: ```shell kubectl apply -f migrate-secret.yaml ``` -- Monitor migration of Secrets using status. Successful migration should have -_Succeeded_ condition set to _true_ in the status field. Get the migration resource +- Monitor migration of Secrets by checking the `.status` of the StorageVersionMigration. + A successful migration should have its +`Succeeded` condition set to true. Get the StorageVersionMigration object as follows: ```shell kubectl get storageversionmigration.storagemigration.k8s.io/secrets-migration -o yaml ``` - This will produce an output similar to: + + The output is similar to: ```yaml kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 @@ -256,20 +257,19 @@ This migration can be achieved through _Storage Version Migration_ to migrate al ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr2 [...] | hexdump -C ``` where [...] contains the additional arguments for connecting to the etcd server. -- Create `StorageVersionMigration` resource file with the name `migrate-crd.yaml` -as follows: +- Create a StorageVersionMigration manifest named `migrate-crd.yaml`, with the contents as follows: ```yaml kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1 metadata: - name: crdsvm + name: crdsvm spec: resource: group: stable.example.com version: v1 resource: SelfieRequest ``` - Create the resource using _kubectl_ as follows: + Create the object using _kubectl_ as follows: ```shell kubectl apply -f migrate-crd.yaml ``` @@ -279,7 +279,8 @@ as follows: ```shell kubectl get storageversionmigration.storagemigration.k8s.io/crdsvm -o yaml ``` - This will produce an output similar to: + + The output is similar to: ```yaml kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1alpha1