From dfbd9db9e3b2ce93e78309bd4d092fc7ec33f143 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 11 Mar 2025 18:51:00 -0700 Subject: [PATCH 01/10] Add design for VolumeGroupSnapshot support Signed-off-by: Shubham Pampattiwar add changelog file Signed-off-by: Shubham Pampattiwar fix codespell checks Signed-off-by: Shubham Pampattiwar address PR feedback: add itemblock:VGS digrams and extra notes for clarification Signed-off-by: Shubham Pampattiwar update backup workflow Signed-off-by: Shubham Pampattiwar --- .../unreleased/8778-shubham-pampattiwar | 1 + design/volume-group-snapshot.md | 605 ++++++++++++++++++ 2 files changed, 606 insertions(+) create mode 100644 changelogs/unreleased/8778-shubham-pampattiwar create mode 100644 design/volume-group-snapshot.md diff --git a/changelogs/unreleased/8778-shubham-pampattiwar b/changelogs/unreleased/8778-shubham-pampattiwar new file mode 100644 index 000000000..569f43ecf --- /dev/null +++ b/changelogs/unreleased/8778-shubham-pampattiwar @@ -0,0 +1 @@ +Add design for VolumeGroupSnapshot support \ No newline at end of file diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md new file mode 100644 index 000000000..b7480e855 --- /dev/null +++ b/design/volume-group-snapshot.md @@ -0,0 +1,605 @@ +# Add Support for VolumeGroupSnapshots + +This proposal outlines the design and implementation plan for incorporating VolumeGroupSnapshot support into Velero. The enhancement will allow Velero to perform consistent, atomic snapshots of groups of Volumes using the new Kubernetes [VolumeGroupSnapshot API](https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/). This capability is especially critical for stateful applications that rely on multiple volumes to ensure data consistency, such as databases and analytics workloads. + +## Background + +Velero currently enables snapshot-based backups on an individual Volume basis through CSI drivers. However, modern stateful applications often require multiple volumes for data, logs, and backups. This distributed data architecture increases the risk of inconsistencies when volumes are captured individually. Kubernetes has introduced the VolumeGroupSnapshot(VGS) API [(KEP-3476)](https://github.com/kubernetes/enhancements/pull/1551), which allows for the atomic snapshotting of multiple volumes in a coordinated manner. By integrating this feature, Velero can offer enhanced disaster recovery for multi-volume applications, ensuring consistency across all related data. + +## Goals +- Ensure that multiple related volumes are snapshotted simultaneously, preserving consistency for stateful applications via VolumeGroupSnapshots(VGS) API. +- Integrate VolumeGroupSnapshot functionality into Velero’s existing backup and restore workflows. +- Allow users to opt in to volume group snapshots via specifying the group label. + +## Non-Goals +- The proposal does not require a complete overhaul of Velero’s CSI integration, it will extend the current mechanism to support group snapshots. +- No any changes pertaining to execution of Restore Hooks + +## High-Level Design + +### Backup workflow: +#### Accept the label to be used for VGS from the user: + - Accept the label from the user, we will do this in 3 ways: + - Firstly, we will have a hard-coded default label key like `velero.io/volume-group-snapshot` that the users can directly use on their PVCs. + - Secondly, we will let the users override this default VGS label via a velero server arg, `--volume-group-nsaphot-label-key`, if needed. + - And Finally we will have the option to override the default label via Backup API spec, `backup.spec.volumeGroupSnapshotLabelKey` + - In all the instances, the VGS label key will be present on the backup spec, this makes the label key accessible to plugins during the execution of backup operation. + - This label will enable velero to filter the PVC to be included in the VGS spec. + - Users will have to label the PVCs before invoking the backup operation. + - This label would act as a group identifier for the PVCs to be grouped under a specific VGS. + - It will be used to collect the PVCs to be used for a particular instance of VGS object. +**Note:** Modifying or adding VGS label on PVCs during an active backup operation may lead to unexpected or undesirable backup results. To avoid inconsistencies, ensure PVC labels remain unchanged throughout the backup execution. + +#### Changes to the Existing PVC ItemBlockAction plugin: + - Currently the PVC IBA plugin is applied to PVCs and adds the RelatedItems for the particular PVC into the ItemBlock. + - At first it checks whether the PVC is bound and VolumeName is non-empty. + - Then it adds the related PV under the list of relatedItems. + - Following on, the plugin adds the pods mounting the PVC as relatedItems. + - Now we need to extend this PVC IBA plugin to add the PVCs to be grouped for a particular VGS object, so that they are processed together under an ItemBlock by Velero. + - First we will check if the PVC that is being processed by the plugin has the user specified VGS label. + - If it is present then we will execute a List call in the namespace with the label as a matching criteria and see if this results in any PVCs (other than the current one). + - If there are PVCs matching the criteria then we add the PVCs to the relatedItems list. + - This helps in building the ItemBlock we need for VGS processing, i.e. we have the relevant pods and PVCs in the ItemBlock. + +**Note:** The ItemBlock to VGS relationship will not always be 1:1. There might be scenarios when the ItemBlock might have multiple VGS instances associated with it. +Lets go ove some ItemBlock/VGS scenarios that we might encounter and visualize them for clarity: +1. Pod Mounts: Pod1 mounts both PVC1 and PVC2. + Grouping: PVC1 and PVC2 share the same group label (group: A) + ItemBlock: The item block includes Pod1, PVC1, and PVC2. + VolumeGroupSnapshot (VGS): Because PVC1 and PVC2 are grouped together by their label, they trigger the creation of a single VGS (labeled with group: A). + +```mermaid +flowchart TD + subgraph ItemBlock + P1[Pod1] + PVC1[PVC1 group: A] + PVC2[PVC2 group: A] + end + + P1 -->|mounts| PVC1 + P1 -->|mounts| PVC2 + + PVC1 --- PVC2 + + PVC1 -- "group: A" --> VGS[VGS group: A] + PVC2 -- "group: A" --> VGS + +``` +2. Pod Mounts: Pod1 mounts each of the four PVCs. + Grouping: + Group A: PVC1 and PVC2 share the same grouping label (group: A). + Group B: PVC3 and PVC4 share the grouping label (group: B) + ItemBlock: All objects (Pod1, PVC1, PVC2, PVC3, and PVC4) are collected into a single item block. + VolumeGroupSnapshots: + PVC1 and PVC2 (group A) point to the same VGS (VGS (group: A)). + PVC3 and PVC4 (group B) point to a different VGS (VGS (group: B)). +```mermaid +flowchart TD + subgraph ItemBlock + P1[Pod1] + PVC1[PVC1 group: A] + PVC2[PVC2 group: A] + PVC3[PVC3 group: B] + PVC4[PVC4 group: B] + end + + %% Pod mounts all PVCs + P1 -->|mounts| PVC1 + P1 -->|mounts| PVC2 + P1 -->|mounts| PVC3 + P1 -->|mounts| PVC4 + + %% Group A relationships: PVC1 and PVC2 + PVC1 --- PVC2 + PVC1 -- "group: A" --> VGS_A[VGS-A group: A] + PVC2 -- "group: A" --> VGS_A + + %% Group B relationships: PVC3 and PVC4 + PVC3 --- PVC4 + PVC3 -- "group: B" --> VGS_B[VGS-B group: B] + PVC4 -- "group: B" --> VGS_B +``` + +3. Pod Mounts: Pod1 mounts both PVC1 and PVC2, Pod2 mounts PVC1 and PVC3. + Grouping: + Group A: PVC1 and PVC2 + Group B: PVC1 and PVC3 + ItemBlock: All objects-Pod1, Pod2, PVC1, PVC2, and PVC3, are collected into a single item block. + VolumeGroupSnapshots: + PVC1 and PVC2 (group A) point to the same VGS (VGS (group: A)). + PVC3 (group B) point to a different VGS (VGS (group: B)). +```mermaid +flowchart TD + subgraph ItemBlock + P1[Pod1] + P2[Pod2] + PVC1[PVC1 group: A, group: B] + PVC2[PVC2 group: A] + PVC3[PVC3 group: B] + end + + %% Pod mount relationships + P1 -->|mounts| PVC1 + P1 -->|mounts| PVC2 + P2 -->|mounts| PVC1 + P2 -->|mounts| PVC3 + + %% Grouping for Group A: PVC1 and PVC2 are grouped into VGS_A + PVC1 --- PVC2 + PVC1 -- "Group A" --> VGS_A[VGS Group A] + PVC2 -- "Group A" --> VGS_A + + %% Grouping for Group B: PVC3 grouped into VGS_B + PVC3 -- "Group B" --> VGS_B[VGS Group B] + +``` + +#### Updates to CSI PVC plugin: + - This is the plugin which creates VolumeSnapshots for the CSI PVCs if required. + - We might encounter the following cases: + - PVC has VGS label and VGS already exists: + - We can check if VGS exists by listing the VGS in the namespace using the backup UUID + along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. + - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label) and also add the PVC names (that are included in the VGS) as + annotations. (This will be useful for our VGS restore operation where we would need PVC resource identifiers). + - For non-datamover case: + - Once VGS is created, add the VGS as an additional item to be included in the backup. + - The VGS BIA plugin will add the related VS as additional items later. + - For datamover case: + - We need to bypass creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. + - Fetch the correct VS instances that got created via VGS and is related to the current PVC. + - Finally create DataUpload object for this VS/PVC pair. + - Add the DataUpload as an additional item. + - PVC has VGS label and VGS exists: + - In this case we do not want to re-create VGS object. + - The required VS/DU for the PVC already exists as VGS instance exists for the PVC and backup UUID. + - PVC does not possess VGS label: + - Create VS and follow the existing legacy workflow. + + +#### Add a new VolumeGroupSnapshot(VGS) BIA plugin + - We need this plugin only for non-datamover case. + - For a particular instance of VGS, this plugin will wait for all the related VS instances to be in ready state. + - And once they are ready, add all the VS instances as additional items for the backup. + - This plugin will also add VolumeGroupSnapshotClass(VGSClass) and VolumeGroupSnapshotContent(VGSC) as additional items to be included in the backup. + +### Add a new VolumeGroupSnapshotClass(VGSClass) BIA plugin + - We need this plugin only for non-datamover case. + - This plugin will be similar to the VSClass plugin. + - This will check if the VGSClass has any lister secret annotations and then add it as an additional item. + +### Add a new VolumeGroupSnapshotContent(VGSC) BIA Plugin + - We need this plugin only for non-datamover case. + - This will also be similar to the VSC plugin + - This will check if the VGSC has any delete secret annotations and then add it as an additional item. + +```mermaid +flowchart TD +%% User Input Section + subgraph User_Input [User Input] + A[User sets VGS label
via default, server arg, or Backup spec] + end + +%% PVC ItemBlockAction Plugin Section + subgraph PVC_IBA [PVC ItemBlockAction Plugin] + B[PVC has VGS label?] + C[If Yes: List all PVCs in namespace
with same label] + D[Add grouped PVCs to relatedItems] + E[Pod IBA adds pods mounting the PVC] + end + +%% CSI PVC Plugin Section + subgraph CSI_PVC [CSI PVC Plugin] + F[Does PVC have VGS label?] + G[If Yes: Check if VGS exists
using backup UID & label] + H[If VGS NOT found,
create new VGS object
backup UID, label, PVC names in annotation] + I[If VGS exists, skip VGS as well as VS creation] + J[If No label: Create individual VS legacy flow] + K[For non-datamover:
Add VGS as additional item] + L[For datamover:
Create DataUpload for VS and add as additional item] + end + +%% VGS BackupItemAction Plugin Section + subgraph VGS_BIA [VGS BackupItemAction Plugin] + M[Wait for all related VS to be ready] + N[Add all ready VS, VGSClass and VGSC as additional items] + end + +%% Existing VS and VSC BIA Plugins + subgraph VS_VSC_BIA [VS and VSC BIA Plugin] + P[Legacy VS and VSC BIA actions] + end + +%% VGSClass and VGSC BIA Plugins + subgraph VSClass_VGSC_BIA [VGSClass and VGSC BIA Plugin] + Q[Similar actions to VSClass and VSC BIA Actions] + end + +%% Flow Connections + A --> B + B -- Yes --> C + C --> D + D --> E + B -- No --> E + + E --> F + F -- Yes --> G + G -- VGS not found --> H + H --> K + G -- VGS exists --> I + F -- No --> J + + K --> VGS_BIA + J -- snapshotMoveData is true --> L + + VGS_BIA --> M + M --> N + N --> P + N --> Q + +``` + + +Restore workflow (WIP): + +- Update the Default Resource Restore priority for Velero restore + - Modify the default priority so that VGS, VGSClass are restored earlier than Pods and PVCs. + +- Add a new VGS RIA plugin + - This plugin will skip restore of the VGS resource. + - It will add the PVCs of the VGS as additional Items. + - The PVC identifiers will be obtained from the VGS annotation that we added during the backup workflow. + + +## Detailed Design + +Backup workflow: +- Accept the label to be used for VGS from the user as a server argument: + - Set a default VGS label key to be used: + ```go + // default VolumeGroupSnapshot Label + defaultVGSLabelKey = "velero.io/volume-group-snapshot" + + ``` + - Add this as a server flag and pass it to backup reconciler, so that we can use it during the backup request execution. + ```go + flags.StringVar(&c.DefaultVGSLabelKey, "volume-group-snapshot-label-key", c.DefaultVGSLabelKey, "Label key for grouping PVCs into VolumeGroupSnapshot") + ``` + + - Update the Backup CRD to accept the VGS Label Key as a spec value: + ```go + // VolumeGroupSnapshotLabelKey specifies the label key to be used for grouping the PVCs under + // an instance of VolumeGroupSnapshot, if left unspecified velero.io/volume-group-snapshot is used + // +optional + VolumeGroupSnapshotLabelKey string `json:"volumeGroupSnapshotLabelKey,omitempty"` + ``` + - Modify the [`prepareBackupRequest` function](https://github.com/openshift/velero/blob/8c8a6cccd78b78bd797e40189b0b9bee46a97f9e/pkg/controller/backup_controller.go#L327) to set the default label key as a backup spec if the user does not specify any value: + ```go + if len(request.Spec.VolumeGroupSnapshotLabelKey) == 0 { + // set the default key value + request.Spec.VolumeGroupSnapshotLabelKey = b.defaultVGSLabelKey + } + ``` + +- Changes to the Existing [PVC ItemBlockAction plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/itemblock/actions/pvc_action.go#L64) (Update the GetRelatedItems function): +```go +// Retrieve the VGS label key from the Backup spec. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + if vgsLabelKey != "" { + // Check if the PVC has the specified VGS label. + if groupID, ok := pvc.Labels[vgsLabelKey]; ok { + // List all PVCs in the namespace with the same label key and value (i.e. same group). + pvcList := new(corev1api.PersistentVolumeClaimList) + if err := a.crClient.List(context.Background(), pvcList, crclient.InNamespace(pvc.Namespace), crclient.MatchingLabels{vgsLabelKey: groupID}); err != nil { + return nil, errors.Wrap(err, "failed to list PVCs for VGS grouping") + } + // Add each matching PVC (except the current one) to the relatedItems. + for _, groupPVC := range pvcList.Items { + if groupPVC.Name == pvc.Name { + continue + } + a.log.Infof("Adding grouped PVC %s to relatedItems for PVC %s", groupPVC.Name, pvc.Name) + relatedItems = append(relatedItems, velero.ResourceIdentifier{ + GroupResource: kuberesource.PersistentVolumeClaims, + Namespace: groupPVC.Namespace, + Name: groupPVC.Name, + }) + } + } + } else { + a.log.Info("No VolumeGroupSnapshotLabelKey provided in backup spec; skipping PVC grouping") + } +``` + +- Updates to [CSI PVC plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/backup/actions/csi/pvc_action.go#L200) (Update the Execute method): +```go +// Retrieve the VGS label key from the backup spec. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + // Check if the PVC has the specified VGS label key. + if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { + // PVC is marked for group snapshot. + existingVGS := p.findExistingVGS(backup.UID, vgsLabelKey, group, pvc.Namespace) + if existingVGS == nil { + // No existing VGS found; list all PVCs in the same group. + groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + // Extract the names of all grouped PVCs. + pvcNames := extractPVCNames(groupedPVCs) + // Create a new VGS object with the backup UID and group. + newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumegroupsnapshots", + }, + Namespace: newVGS.Namespace, + Name: newVGS.Name, + }) + } else { + // Existing VGS found; skip creating an individual VS. + p.log.Infof("PVC %s is part of existing VGS %s, skipping individual VolumeSnapshot creation", pvc.Name, existingVGS.Name) + } + } else { + // PVC is not part of a VGS group; create an individual VolumeSnapshot. + + //. + //. + //. + } + +// helper functions used + +// findExistingVGS checks for an existing VolumeGroupSnapshot (VGS) for the given backup UID and group in the namespace. +func (p *pvcBackupItemAction) findExistingVGS(backupUID types.UID, vgsLabelKey, group, namespace string) *VolumeGroupSnapshot { + var vgsList VolumeGroupSnapshotList + err := p.crClient.List( + context.TODO(), + &vgsList, + crclient.InNamespace(namespace), + crclient.MatchingLabels{ + "backup-uid": string(backupUID), + vgsLabelKey: group, + }, + ) + if err != nil { + p.log.Errorf("Error listing VGS: %v", err) + return nil + } + if len(vgsList.Items) > 0 { + return &vgsList.Items[0] + } + return nil +} + +// listGroupedPVCs returns all PVCs in the given namespace that have the specified VGS label key and group value. +func (p *pvcBackupItemAction) listGroupedPVCs(backup *velerov1api.Backup, namespace, vgsLabelKey, group string) ([]corev1api.PersistentVolumeClaim, error) { + var pvcList corev1api.PersistentVolumeClaimList + err := p.crClient.List(context.TODO(), &pvcList, crclient.InNamespace(namespace), crclient.MatchingLabels{vgsLabelKey: group}) + if err != nil { + return nil, err + } + return pvcList.Items, nil +} + +func (p *pvcBackupItemAction) createVolumeGroupSnapshot( + backup *velerov1api.Backup, + pvc corev1api.PersistentVolumeClaim, + pvcNames []string, + vgsLabelKey, group string, +) (*VolumeGroupSnapshot, error) { + // Generate a name for the new VGS object. + name := fmt.Sprintf("vgs-%s-%d", string(backup.UID), time.Now().Unix()) + + // Construct the VGS object + vgs := &VolumeGroupSnapshot{ + TypeMeta: metav1.TypeMeta{ + Kind: "VolumeGroupSnapshot", + APIVersion: "groupsnapshot.storage.k8s.io/v1beta1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: pvc.Namespace, + Annotations: map[string]string{ + "pvcList": strings.Join(pvcNames, ","), + }, + Labels: map[string]string{ + velerov1api.BackupUIDLabel: string(backup.UID), + vgsLabelKey: group, + }, + }, + Spec: VolumeGroupSnapshotSpec{ + // we will use default VGSClass for now + Source: VolumeGroupSnapshotSource{ + Selector: metav1.LabelSelector{ + MatchLabels: map[string]string{ + vgsLabelKey: group, + }, + }, + }, + }, + } + + // Create the VGS object + if err := p.crClient.Create(context.TODO(), vgs); err != nil { + return nil, err + } + return vgs, nil +} + +// extractPVCNames returns a slice of PVC names from the provided list. +func extractPVCNames(pvcs []corev1api.PersistentVolumeClaim) []string { + names := []string{} + for _, pvc := range pvcs { + names = append(names, pvc.Name) + } + return names +} + +``` + +- Add a new VolumeGroupSnapshot(VGS) BIA plugin +```go +// AppliesTo specifies that this plugin applies to VolumeGroupSnapshot objects. +func (p *vgsBackupItemAction) AppliesTo() (velero.ResourceSelector, error) { + return velero.ResourceSelector{ + IncludedResources: []string{"volumegroupsnapshots"}, + }, nil +} + +// Execute waits for all related VolumeSnapshot (VS) instances to be ready, then adds them +// along with the VolumeGroupSnapshotClass as additional backup items. +func (p *vgsBackupItemAction) Execute( + item runtime.Unstructured, + backup *velerov1api.Backup, +) ( + runtime.Unstructured, + []velero.ResourceIdentifier, + string, + []velero.ResourceIdentifier, + error, +) { + p.log.Infof("Starting VGS backup plugin Execute for item: %s", item.GetName()) + + // Convert the unstructured object to a VolumeGroupSnapshot. + var vgs VolumeGroupSnapshot + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(item.UnstructuredContent(), &vgs); err != nil { + return nil, nil, "", nil, errors.Wrap(err, "unable to convert item to VolumeGroupSnapshot") + } + + // Wait until all related VolumeSnapshots are ready. + p.log.Infof("Waiting for all related VolumeSnapshots for VGS %s to be ready", vgs.Name) + ready, err := waitForGroupedVolumeSnapshots(&vgs, vgs.Namespace, p.log) + if err != nil { + return nil, nil, "", nil, errors.Wrap(err, "error waiting for grouped VolumeSnapshots") + } + if !ready { + p.log.Infof("Not all related VolumeSnapshots are ready for VGS %s", vgs.Name) + return item, nil, "", nil, nil + } + + // Retrieve the related VolumeSnapshot instances. + groupedVS := getGroupedVolumeSnapshots(&vgs, vgs.Namespace) + var additionalItems []velero.ResourceIdentifier + for _, vs := range groupedVS { + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumesnapshots", + }, + Namespace: vs.Namespace, + Name: vs.Name, + }) + } + + // Add the VolumeGroupSnapshotClass as an additional item. + // Here we assume that the VGS spec defines the class name. + // We will be using the default VGSClass for now + vgsClassName := vgs.Spec.VolumeGroupSnapshotClassName + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumegroupsnapshotclasses", + }, + Namespace: "", + Name: vgsClassName, + }) + + p.log.Infof("VGS %s ready; adding %d additional items to backup", vgs.Name, len(additionalItems)) + return item, additionalItems, "", nil, nil +} +``` + +Restore workflow: + +- Update the [Default Resource Restore priority](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/cmd/server/config/config.go#L116) for Velero restore +```go +defaultRestorePriorities = types.Priorities{ + HighPriorities: []string{ + "customresourcedefinitions", + "namespaces", + "storageclasses", + // updated list from here + "volumegroupsnaoshotclass.snapshot.k8s.io", + "volumegroupsnapshots.snapshot.storage.k8s.io", + "volumesnapshotclass.snapshot.storage.k8s.io", + "volumesnapshotcontents.snapshot.storage.k8s.io", + "volumesnapshots.snapshot.storage.k8s.io", + "datauploads.velero.io", + "persistentvolumes", + "persistentvolumeclaims", +``` +- Add a VGS RIA plugin +```go + +// AppliesTo returns a selector that matches VolumeGroupSnapshot resources. +func (p *vgsRestoreItemAction) AppliesTo() (velero.ResourceSelector, error) { + return velero.ResourceSelector{ + IncludedResources: []string{"volumegroupsnapshots"}, + }, nil +} + + +// Execute skips restoring the VGS resource and extracts the PVC identifiers from the "pvcList" annotation, +// adding each PVC as an additional restore item. +func (p *vgsRestoreItemAction) Execute( + input *velero.RestoreItemActionExecuteInput, +) (*velero.RestoreItemActionExecuteOutput, error) { + p.log.Info("Starting VGS RestoreItemAction") + + // Convert the backup VGS object to an unstructured type. + // We use input.ItemFromBackup as the backed-up version of the VGS. + var vgs unstructured.Unstructured + vgs.SetUnstructuredContent(input.ItemFromBackup.UnstructuredContent()) + + // Retrieve the pvcList annotation. + annotations := vgs.GetAnnotations() + pvcListStr, ok := annotations["pvcList"] + if !ok || pvcListStr == "" { + p.log.Infof("No pvcList annotation found on VGS %s", vgs.GetName()) + return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil + } + + // Parse the comma-separated list of PVC names. + pvcNames := strings.Split(pvcListStr, ",") + var additionalItems []velero.ResourceIdentifier + namespace := vgs.GetNamespace() + for _, pvcName := range pvcNames { + pvcName = strings.TrimSpace(pvcName) + if pvcName == "" { + continue + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "", + Resource: "persistentvolumeclaims", + }, + Namespace: namespace, + Name: pvcName, + }) + } + + p.log.Infof("Skipping restore of VGS %s and adding %d PVC(s) as additional restore items", vgs.GetName(), len(additionalItems)) + + // Return output indicating that the VGS resource itself should be skipped, + // and providing the list of PVCs to be restored as additional items + return &velero.RestoreItemActionExecuteOutput{ + SkipRestore: true, + AdditionalItems: additionalItems, + }, nil +} +``` +## Implementation + +This design proposal is targeted for velero 1.16. + +The implementation of this proposed design is targeted for velero 1.17. + +## Open Questions + +How to handle VGSC ? Do we need VGSC for non-datamover operations ? From 48d6aff78606338d2c274a9e4e7d992da0292332 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Mon, 24 Mar 2025 12:45:36 -0700 Subject: [PATCH 02/10] update itemblock case 3 Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index b7480e855..1eb23f131 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -103,7 +103,7 @@ flowchart TD 3. Pod Mounts: Pod1 mounts both PVC1 and PVC2, Pod2 mounts PVC1 and PVC3. Grouping: Group A: PVC1 and PVC2 - Group B: PVC1 and PVC3 + Group B: PVC3 ItemBlock: All objects-Pod1, Pod2, PVC1, PVC2, and PVC3, are collected into a single item block. VolumeGroupSnapshots: PVC1 and PVC2 (group A) point to the same VGS (VGS (group: A)). @@ -113,7 +113,7 @@ flowchart TD subgraph ItemBlock P1[Pod1] P2[Pod2] - PVC1[PVC1 group: A, group: B] + PVC1[PVC1 group: A] PVC2[PVC2 group: A] PVC3[PVC3 group: B] end From 0c87e2f64d4618a99aa15feab8c6e50a0303b04c Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 25 Mar 2025 15:03:34 -0700 Subject: [PATCH 03/10] Update the VGS B/R workflows Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 229 +++++++++----------------------- 1 file changed, 64 insertions(+), 165 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 1eb23f131..ca5f1ca83 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -42,7 +42,7 @@ Velero currently enables snapshot-based backups on an individual Volume basis th - This helps in building the ItemBlock we need for VGS processing, i.e. we have the relevant pods and PVCs in the ItemBlock. **Note:** The ItemBlock to VGS relationship will not always be 1:1. There might be scenarios when the ItemBlock might have multiple VGS instances associated with it. -Lets go ove some ItemBlock/VGS scenarios that we might encounter and visualize them for clarity: +Lets go over some ItemBlock/VGS scenarios that we might encounter and visualize them for clarity: 1. Pod Mounts: Pod1 mounts both PVC1 and PVC2. Grouping: PVC1 and PVC2 share the same group label (group: A) ItemBlock: The item block includes Pod1, PVC1, and PVC2. @@ -137,119 +137,97 @@ flowchart TD #### Updates to CSI PVC plugin: - This is the plugin which creates VolumeSnapshots for the CSI PVCs if required. - We might encounter the following cases: - - PVC has VGS label and VGS already exists: + - PVC has VGS label and VGS does not exist: - We can check if VGS exists by listing the VGS in the namespace using the backup UUID along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label) and also add the PVC names (that are included in the VGS) as - annotations. (This will be useful for our VGS restore operation where we would need PVC resource identifiers). + annotations. + - Once VGS is created, wait for VGS object to be `readyToUse`, it implies that all the related VS/VSC objects are ready to use too. + - Add the VGS object as an additional item. - For non-datamover case: - - Once VGS is created, add the VGS as an additional item to be included in the backup. - - The VGS BIA plugin will add the related VS as additional items later. + - Here, we need to skip the creation of VS as the VS object get created via creation of VGS object. + - We just need to add the VS objects that got created via VGS object as additional items and the rest of the workflow remains the same for non-datamover cases. - For datamover case: - - We need to bypass creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. + - We need to skip creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. - Fetch the correct VS instances that got created via VGS and is related to the current PVC. - Finally create DataUpload object for this VS/PVC pair. - Add the DataUpload as an additional item. - - PVC has VGS label and VGS exists: + - PVC has VGS label and VGS already exists: - In this case we do not want to re-create VGS object. - The required VS/DU for the PVC already exists as VGS instance exists for the PVC and backup UUID. - PVC does not possess VGS label: - Create VS and follow the existing legacy workflow. - #### Add a new VolumeGroupSnapshot(VGS) BIA plugin - - We need this plugin only for non-datamover case. - - For a particular instance of VGS, this plugin will wait for all the related VS instances to be in ready state. - - And once they are ready, add all the VS instances as additional items for the backup. - - This plugin will also add VolumeGroupSnapshotClass(VGSClass) and VolumeGroupSnapshotContent(VGSC) as additional items to be included in the backup. - -### Add a new VolumeGroupSnapshotClass(VGSClass) BIA plugin - - We need this plugin only for non-datamover case. - - This plugin will be similar to the VSClass plugin. - - This will check if the VGSClass has any lister secret annotations and then add it as an additional item. - -### Add a new VolumeGroupSnapshotContent(VGSC) BIA Plugin - - We need this plugin only for non-datamover case. - - This will also be similar to the VSC plugin - - This will check if the VGSC has any delete secret annotations and then add it as an additional item. + - Cleanup the VGS object, here we need to be careful, we want to delete VGS and VGSC objects but keep the VS and VSC objects for further processing of our backup. + - We will first check the `deletionPolicy` of VGSC, + - if its `Retain` then we can go ahead with the deletion of VGS and VGSC objects. + - But if its `Delete` then we need to first patch the VGSC `deletionPolicy` to `Retain` and then delete the VGS and VGSC objects. ```mermaid flowchart TD -%% User Input Section - subgraph User_Input [User Input] - A[User sets VGS label
via default, server arg, or Backup spec] +%% User Input + subgraph "User Input" + A[User sets VGS label key
default, server arg, or Backup spec] + B[User labels PVCs before backup] + A --> B end -%% PVC ItemBlockAction Plugin Section - subgraph PVC_IBA [PVC ItemBlockAction Plugin] - B[PVC has VGS label?] - C[If Yes: List all PVCs in namespace
with same label] - D[Add grouped PVCs to relatedItems] - E[Pod IBA adds pods mounting the PVC] +%% PVC ItemBlockAction Plugin + subgraph "PVC IBA Plugin" + C[Check: PVC is bound & VolumeName non-empty] + D[Add related PV and pods] + E[Check if PVC has VGS label] + F[List all PVCs with same label in namespace] + G[Add PVCs to relatedItems] + C --> D + D --> E + E -- Yes --> F + F --> G end -%% CSI PVC Plugin Section - subgraph CSI_PVC [CSI PVC Plugin] - F[Does PVC have VGS label?] - G[If Yes: Check if VGS exists
using backup UID & label] - H[If VGS NOT found,
create new VGS object
backup UID, label, PVC names in annotation] - I[If VGS exists, skip VGS as well as VS creation] - J[If No label: Create individual VS legacy flow] - K[For non-datamover:
Add VGS as additional item] - L[For datamover:
Create DataUpload for VS and add as additional item] - end +%% CSI PVC Plugin +subgraph "CSI PVC Plugin" +H[For each PVC, check for VGS label] +I[If VGS label exists:] +J[Lookup VGS using backup UID and label] +K[Create VGS object] +L[Wait until VGS is readyToUse, add VGS as an additional item] +M[Non-datamover: Skip individual VS creation; add VS objects from VGS as additional items] +N[Datamover: Create DataUpload for VS PVC pair; add DU as an additional item] +O[If VGS already exists, skip creation] +P[If no VGS label, follow legacy workflow: create VS] +H --> I +I -- Yes --> J +J -- Not found --> K +K --> L +L --> M +L --> N +I -- Yes and found --> O +I -- No --> P +end -%% VGS BackupItemAction Plugin Section - subgraph VGS_BIA [VGS BackupItemAction Plugin] - M[Wait for all related VS to be ready] - N[Add all ready VS, VGSClass and VGSC as additional items] - end +%% VGS BackupItemAction Plugin +subgraph "VGS BIA Plugin" +Q[Check VGSC deletionPolicy] +R[If Retain, delete VGS and VGSC objects] +S[If Delete, patch VGSC to Retain then delete VGS and VGSC] +Q --> R +Q --> S +end -%% Existing VS and VSC BIA Plugins - subgraph VS_VSC_BIA [VS and VSC BIA Plugin] - P[Legacy VS and VSC BIA actions] - end - -%% VGSClass and VGSC BIA Plugins - subgraph VSClass_VGSC_BIA [VGSClass and VGSC BIA Plugin] - Q[Similar actions to VSClass and VSC BIA Actions] - end - -%% Flow Connections - A --> B - B -- Yes --> C - C --> D - D --> E - B -- No --> E - - E --> F - F -- Yes --> G - G -- VGS not found --> H - H --> K - G -- VGS exists --> I - F -- No --> J - - K --> VGS_BIA - J -- snapshotMoveData is true --> L - - VGS_BIA --> M - M --> N - N --> P - N --> Q +%% Connect the flows +B --> C +G --> H +M --> Q +N --> Q ``` -Restore workflow (WIP): - -- Update the Default Resource Restore priority for Velero restore - - Modify the default priority so that VGS, VGSClass are restored earlier than Pods and PVCs. - -- Add a new VGS RIA plugin - - This plugin will skip restore of the VGS resource. - - It will add the PVCs of the VGS as additional Items. - - The PVC identifiers will be obtained from the VGS annotation that we added during the backup workflow. +Restore workflow: +No changes required for the restore workflow. ## Detailed Design @@ -515,85 +493,6 @@ func (p *vgsBackupItemAction) Execute( } ``` -Restore workflow: - -- Update the [Default Resource Restore priority](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/cmd/server/config/config.go#L116) for Velero restore -```go -defaultRestorePriorities = types.Priorities{ - HighPriorities: []string{ - "customresourcedefinitions", - "namespaces", - "storageclasses", - // updated list from here - "volumegroupsnaoshotclass.snapshot.k8s.io", - "volumegroupsnapshots.snapshot.storage.k8s.io", - "volumesnapshotclass.snapshot.storage.k8s.io", - "volumesnapshotcontents.snapshot.storage.k8s.io", - "volumesnapshots.snapshot.storage.k8s.io", - "datauploads.velero.io", - "persistentvolumes", - "persistentvolumeclaims", -``` -- Add a VGS RIA plugin -```go - -// AppliesTo returns a selector that matches VolumeGroupSnapshot resources. -func (p *vgsRestoreItemAction) AppliesTo() (velero.ResourceSelector, error) { - return velero.ResourceSelector{ - IncludedResources: []string{"volumegroupsnapshots"}, - }, nil -} - - -// Execute skips restoring the VGS resource and extracts the PVC identifiers from the "pvcList" annotation, -// adding each PVC as an additional restore item. -func (p *vgsRestoreItemAction) Execute( - input *velero.RestoreItemActionExecuteInput, -) (*velero.RestoreItemActionExecuteOutput, error) { - p.log.Info("Starting VGS RestoreItemAction") - - // Convert the backup VGS object to an unstructured type. - // We use input.ItemFromBackup as the backed-up version of the VGS. - var vgs unstructured.Unstructured - vgs.SetUnstructuredContent(input.ItemFromBackup.UnstructuredContent()) - - // Retrieve the pvcList annotation. - annotations := vgs.GetAnnotations() - pvcListStr, ok := annotations["pvcList"] - if !ok || pvcListStr == "" { - p.log.Infof("No pvcList annotation found on VGS %s", vgs.GetName()) - return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil - } - - // Parse the comma-separated list of PVC names. - pvcNames := strings.Split(pvcListStr, ",") - var additionalItems []velero.ResourceIdentifier - namespace := vgs.GetNamespace() - for _, pvcName := range pvcNames { - pvcName = strings.TrimSpace(pvcName) - if pvcName == "" { - continue - } - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "", - Resource: "persistentvolumeclaims", - }, - Namespace: namespace, - Name: pvcName, - }) - } - - p.log.Infof("Skipping restore of VGS %s and adding %d PVC(s) as additional restore items", vgs.GetName(), len(additionalItems)) - - // Return output indicating that the VGS resource itself should be skipped, - // and providing the list of PVCs to be restored as additional items - return &velero.RestoreItemActionExecuteOutput{ - SkipRestore: true, - AdditionalItems: additionalItems, - }, nil -} -``` ## Implementation This design proposal is targeted for velero 1.16. From 5ce4b5ad643a855a4fd84765fb040f93f0a99dfb Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 27 Mar 2025 11:28:36 -0700 Subject: [PATCH 04/10] remove vgsc open question Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index ca5f1ca83..47f06acfc 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -499,6 +499,3 @@ This design proposal is targeted for velero 1.16. The implementation of this proposed design is targeted for velero 1.17. -## Open Questions - -How to handle VGSC ? Do we need VGSC for non-datamover operations ? From d4296aa78c963e366d3b983b95572c3cd11b9340 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 27 Mar 2025 18:18:29 -0700 Subject: [PATCH 05/10] delegate cleanup to VGS BIA Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 116 ++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 52 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 47f06acfc..6b9bd0dd5 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -140,17 +140,18 @@ flowchart TD - PVC has VGS label and VGS does not exist: - We can check if VGS exists by listing the VGS in the namespace using the backup UUID along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. - - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label) and also add the PVC names (that are included in the VGS) as - annotations. - - Once VGS is created, wait for VGS object to be `readyToUse`, it implies that all the related VS/VSC objects are ready to use too. - - Add the VGS object as an additional item. + - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label). + - Once VGS is created, we just wait for the related VS/VSC objects to exist and not wait for them to be `readyToUse`. + - Add the VGS object to the list of itemToUpdate, so that its BIA plugin runs in finalize backup phase. (we will run all the cleanup tasks in this plugin) - For non-datamover case: - Here, we need to skip the creation of VS as the VS object get created via creation of VGS object. + - We will skip the cleanup of VS here in VS BIA plugin as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - We just need to add the VS objects that got created via VGS object as additional items and the rest of the workflow remains the same for non-datamover cases. - For datamover case: - We need to skip creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. - Fetch the correct VS instances that got created via VGS and is related to the current PVC. - Finally create DataUpload object for this VS/PVC pair. + - We will skip the cleanup of VS here as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - Add the DataUpload as an additional item. - PVC has VGS label and VGS already exists: - In this case we do not want to re-create VGS object. @@ -158,76 +159,87 @@ flowchart TD - PVC does not possess VGS label: - Create VS and follow the existing legacy workflow. +#### Modify the CleanupVolumeSnapshot function to skip deletion of VGS related VS +- We use a common cleanupVolumeSnapshot function to cleanup VS objects in both datamover and non-datamover cases +- We are delegating the VGS related cleanup tasks to VGS BIA plugin which will run during the backup finalize phase when all the async actions are complete +- We need to modify this function skip deletion of VS objects that were created via VGS objects. + #### Add a new VolumeGroupSnapshot(VGS) BIA plugin - - Cleanup the VGS object, here we need to be careful, we want to delete VGS and VGSC objects but keep the VS and VSC objects for further processing of our backup. - - We will first check the `deletionPolicy` of VGSC, - - if its `Retain` then we can go ahead with the deletion of VGS and VGSC objects. - - But if its `Delete` then we need to first patch the VGSC `deletionPolicy` to `Retain` and then delete the VGS and VGSC objects. + - This plugin will run in backup finalize phase. + - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created durring the backup process. ```mermaid flowchart TD -%% User Input +%% User Input Section subgraph "User Input" - A[User sets VGS label key
default, server arg, or Backup spec] - B[User labels PVCs before backup] - A --> B + U1[User sets VGS label key using default, server arg or Backup API spec] + U2[User labels PVCs before backup] + U1 --> U2 end -%% PVC ItemBlockAction Plugin +%% PVC ItemBlockAction Plugin Section subgraph "PVC IBA Plugin" - C[Check: PVC is bound & VolumeName non-empty] - D[Add related PV and pods] - E[Check if PVC has VGS label] - F[List all PVCs with same label in namespace] - G[Add PVCs to relatedItems] - C --> D - D --> E - E -- Yes --> F - F --> G + I1[Check PVC is bound and has VolumeName] + I2[Add related PV to relatedItems] + I3[Add pods mounting PVC to relatedItems] + I4[Check if PVC has user-specified VGS label] + I5[List PVCs in namespace matching label criteria] + I6[Add matching PVCs to relatedItems] + I1 --> I2 + I2 --> I3 + I3 --> I4 + I4 -- Yes --> I5 + I5 --> I6 end -%% CSI PVC Plugin +%% CSI PVC Plugin Section subgraph "CSI PVC Plugin" -H[For each PVC, check for VGS label] -I[If VGS label exists:] -J[Lookup VGS using backup UID and label] -K[Create VGS object] -L[Wait until VGS is readyToUse, add VGS as an additional item] -M[Non-datamover: Skip individual VS creation; add VS objects from VGS as additional items] -N[Datamover: Create DataUpload for VS PVC pair; add DU as an additional item] -O[If VGS already exists, skip creation] -P[If no VGS label, follow legacy workflow: create VS] -H --> I -I -- Yes --> J -J -- Not found --> K -K --> L -L --> M -L --> N -I -- Yes and found --> O -I -- No --> P +C1[For each PVC, check for VGS label] +C2[If VGS label exists then lookup VGS using backup UID and label] +C3[If VGS not found then create new VGS with backup UID and group label] +C4[Wait for related VS and VGSC objects to exist] +C5[Add VGS object to itemToUpdate] +C6[Non-datamover case: Skip individual VS creation; add VS from VGS as additional item] +C7[Datamover case: Create DataUpload for VS-PVC pair; add DU as additional item] +C8[If VGS already exists then do not re-create VGS] +C9[If no VGS label then follow legacy workflow to create VS] +C1 --> C2 +C2 -- Not Found --> C3 +C3 --> C4 +C4 --> C5 +C5 --> C6 +C2 -- Found --> C8 +C1 -- No VGS label --> C9 end -%% VGS BackupItemAction Plugin +%% Cleanup Modification Section +subgraph "Cleanup Modification" +CL1[Modify cleanupVolumeSnapshot to skip deletion of VS created via VGS] +CL2[Delegate VGS related cleanup to VGS BIA Plugin in finalize phase] +CL1 --> CL2 +end + +%% VGS BackupItemAction Plugin Section subgraph "VGS BIA Plugin" -Q[Check VGSC deletionPolicy] -R[If Retain, delete VGS and VGSC objects] -S[If Delete, patch VGSC to Retain then delete VGS and VGSC] -Q --> R -Q --> S +V1[Run in backup finalize phase] +V2[Process cleanup of VGS, VGSC, VS and VSC artifacts] +V1 --> V2 end -%% Connect the flows -B --> C -G --> H -M --> Q -N --> Q +%% Connect Major Sections +U2 --> I1 +I6 --> C1 +C6 --> CL1 +C7 --> CL1 +CL2 --> V1 ``` Restore workflow: -No changes required for the restore workflow. +#### Add a new VolumeGroupSnapshot(VGS) RIA plugin +- This plugin will just skip the restore VGS object ## Detailed Design From e9f23a32eef2f7576d332cbcf9a4384733b1e083 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 27 Mar 2025 19:00:34 -0700 Subject: [PATCH 06/10] fix typo Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 6b9bd0dd5..f4d0dc53a 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -166,7 +166,7 @@ flowchart TD #### Add a new VolumeGroupSnapshot(VGS) BIA plugin - This plugin will run in backup finalize phase. - - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created durring the backup process. + - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created during the backup process. ```mermaid flowchart TD From 0ab2253f46752741742db09daea9543607993ada Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 3 Apr 2025 13:35:49 -0700 Subject: [PATCH 07/10] update csi plugin changes, diagram and snippets Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 548 ++++++++++++++++---------------- 1 file changed, 278 insertions(+), 270 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index f4d0dc53a..71e47d2ec 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -135,111 +135,85 @@ flowchart TD ``` #### Updates to CSI PVC plugin: - - This is the plugin which creates VolumeSnapshots for the CSI PVCs if required. - - We might encounter the following cases: - - PVC has VGS label and VGS does not exist: - - We can check if VGS exists by listing the VGS in the namespace using the backup UUID - along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. - - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label). - - Once VGS is created, we just wait for the related VS/VSC objects to exist and not wait for them to be `readyToUse`. - - Add the VGS object to the list of itemToUpdate, so that its BIA plugin runs in finalize backup phase. (we will run all the cleanup tasks in this plugin) - - For non-datamover case: - - Here, we need to skip the creation of VS as the VS object get created via creation of VGS object. - - We will skip the cleanup of VS here in VS BIA plugin as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - - We just need to add the VS objects that got created via VGS object as additional items and the rest of the workflow remains the same for non-datamover cases. - - For datamover case: - - We need to skip creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. - - Fetch the correct VS instances that got created via VGS and is related to the current PVC. - - Finally create DataUpload object for this VS/PVC pair. - - We will skip the cleanup of VS here as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - - Add the DataUpload as an additional item. - - PVC has VGS label and VGS already exists: - - In this case we do not want to re-create VGS object. - - The required VS/DU for the PVC already exists as VGS instance exists for the PVC and backup UUID. - - PVC does not possess VGS label: - - Create VS and follow the existing legacy workflow. +- When a PVC has a VGS label and no VS (created via VGS) exists: + - Create VGS: + - This triggers creation of the corresponding VGSC, VS, and VSC objects. + - Wait for VS Status: + - Wait until each VS (one per PVC in the group) has its volumeGroupSnapshotName set. This confirms that the snapshot controller has done its work. + - Update VS Objects: + - Remove owner references and VGS-related finalizers from the VS objects (decoupling them to prevent cascading deletion). + - Add backup metadata (BackupName, BackupUUID, PVC name) as labels. This metadata is later used to skip re-creating a VGS when another PVC of the same group is processed. + - Patch and Cleanup: + - Patch the VGSC deletionPolicy to Retain so that when you delete the VGSC, the underlying VSC (and the storage snapshots) remain. + - Delete the temporary VGS and VGSC objects. + - Branching: + - For non‑datamover cases, skip the creation of an individual VS (since it was created via VGS) and add the VS objects as additional items. + - For datamover cases, create DataUploads for the VS–PVC pair (using the VS created by the VGS workflow) and add those as additional items. + +- When a PVC has a VGS label and a VS created via an earlier VGS workflow already exists: + - List VS objects in the PVC’s namespace using labels (BackupUUID, BackupName, PVCName). + - Verify that a VS exists and that its status shows a non‑empty volumeGroupSnapshotName. + - If so, skip VGS (and VS) creation and continue with the legacy workflow. + - If a VS is found but it wasn’t created by the VGS workflow (i.e. it lacks the volumeGroupSnapshotName), then the backup for that PVC is failed, resulting in a partially failed backup. -#### Modify the CleanupVolumeSnapshot function to skip deletion of VGS related VS -- We use a common cleanupVolumeSnapshot function to cleanup VS objects in both datamover and non-datamover cases -- We are delegating the VGS related cleanup tasks to VGS BIA plugin which will run during the backup finalize phase when all the async actions are complete -- We need to modify this function skip deletion of VS objects that were created via VGS objects. +- When a PVC does not have a VGS label: + - The legacy workflow is followed, creating an individual VolumeSnapshot as before. -#### Add a new VolumeGroupSnapshot(VGS) BIA plugin - - This plugin will run in backup finalize phase. - - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created during the backup process. ```mermaid flowchart TD -%% User Input Section - subgraph "User Input" - U1[User sets VGS label key using default, server arg or Backup API spec] - U2[User labels PVCs before backup] - U1 --> U2 +%% Section 1: Accept VGS Label from User + subgraph Accept_Label + A1[User sets VGS label] + A2[User labels PVCs before backup] + A1 --> A2 end -%% PVC ItemBlockAction Plugin Section - subgraph "PVC IBA Plugin" - I1[Check PVC is bound and has VolumeName] - I2[Add related PV to relatedItems] - I3[Add pods mounting PVC to relatedItems] - I4[Check if PVC has user-specified VGS label] - I5[List PVCs in namespace matching label criteria] - I6[Add matching PVCs to relatedItems] - I1 --> I2 - I2 --> I3 - I3 --> I4 - I4 -- Yes --> I5 - I5 --> I6 +%% Section 2: PVC ItemBlockAction Plugin + subgraph PVC_ItemBlockAction + B1[Check PVC is bound and has VolumeName] + B2[Add related PV to relatedItems] + B3[Add pods mounting PVC to relatedItems] + B4[Check if PVC has user-specified VGS label] + B5[List PVCs in namespace matching label criteria] + B6[Add matching PVCs to relatedItems] + B1 --> B2 --> B3 --> B4 + B4 -- Yes --> B5 + B5 --> B6 end -%% CSI PVC Plugin Section -subgraph "CSI PVC Plugin" +%% Section 3: CSI PVC Plugin Updates +subgraph CSI_PVC_Plugin C1[For each PVC, check for VGS label] -C2[If VGS label exists then lookup VGS using backup UID and label] -C3[If VGS not found then create new VGS with backup UID and group label] -C4[Wait for related VS and VGSC objects to exist] -C5[Add VGS object to itemToUpdate] -C6[Non-datamover case: Skip individual VS creation; add VS from VGS as additional item] -C7[Datamover case: Create DataUpload for VS-PVC pair; add DU as additional item] -C8[If VGS already exists then do not re-create VGS] -C9[If no VGS label then follow legacy workflow to create VS] -C1 --> C2 -C2 -- Not Found --> C3 -C3 --> C4 -C4 --> C5 -C5 --> C6 -C2 -- Found --> C8 -C1 -- No VGS label --> C9 +C1 -- Yes --> C2[Case 1: VGS for volume group not yet created] +C2 -- True --> C3[Create new VGS triggering VGSC, VS and VSC creation] +C3 --> C4[Wait for VS objects to show volumeGroupSnapshotName using CSISnapshotTimeout] +C4 --> C5[Update VS objects: remove VGS owner refs and VGS finalizers; add BackupName, BackupUUID, PVC name as labels] +C5 --> C6[Patch VGSC deletionPolicy to Retain] +C6 --> C7[Delete VGS object] +C7 --> C8[Delete VGSC] +C8 --> C9[For non-datamover: Skip individual VS creation; add VS from VGS as additional item] +C8 --> C10[For datamover: Create DataUpload for VS-PVC pair; add DU as additional item] + +C1 -- Yes --> C11[Case 2: VGS was created then deleted] +C11 --> C12[List VS using labels BackupUID, BackupName, PVCName] +C12 --> C13[Check if VS has non-empty volumeGroupSnapshotName] +C13 -- Yes --> C14[Skip VGS creation; use existing VS/DU; legacy workflow continues] +C13 -- No --> C15[Fail backup for PVC] + +C1 -- No --> C16[Case 3: PVC lacks VGS label; follow legacy workflow to create VS] end -%% Cleanup Modification Section -subgraph "Cleanup Modification" -CL1[Modify cleanupVolumeSnapshot to skip deletion of VS created via VGS] -CL2[Delegate VGS related cleanup to VGS BIA Plugin in finalize phase] -CL1 --> CL2 -end - -%% VGS BackupItemAction Plugin Section -subgraph "VGS BIA Plugin" -V1[Run in backup finalize phase] -V2[Process cleanup of VGS, VGSC, VS and VSC artifacts] -V1 --> V2 -end - -%% Connect Major Sections -U2 --> I1 -I6 --> C1 -C6 --> CL1 -C7 --> CL1 -CL2 --> V1 +%% Connect Main Sections +A2 --> B1 +B6 --> C1 ``` Restore workflow: -#### Add a new VolumeGroupSnapshot(VGS) RIA plugin -- This plugin will just skip the restore VGS object +- No changes required for the restore workflow. ## Detailed Design @@ -303,148 +277,7 @@ Backup workflow: - Updates to [CSI PVC plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/backup/actions/csi/pvc_action.go#L200) (Update the Execute method): ```go -// Retrieve the VGS label key from the backup spec. - vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey - // Check if the PVC has the specified VGS label key. - if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { - // PVC is marked for group snapshot. - existingVGS := p.findExistingVGS(backup.UID, vgsLabelKey, group, pvc.Namespace) - if existingVGS == nil { - // No existing VGS found; list all PVCs in the same group. - groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - // Extract the names of all grouped PVCs. - pvcNames := extractPVCNames(groupedPVCs) - // Create a new VGS object with the backup UID and group. - newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumegroupsnapshots", - }, - Namespace: newVGS.Namespace, - Name: newVGS.Name, - }) - } else { - // Existing VGS found; skip creating an individual VS. - p.log.Infof("PVC %s is part of existing VGS %s, skipping individual VolumeSnapshot creation", pvc.Name, existingVGS.Name) - } - } else { - // PVC is not part of a VGS group; create an individual VolumeSnapshot. - - //. - //. - //. - } - -// helper functions used - -// findExistingVGS checks for an existing VolumeGroupSnapshot (VGS) for the given backup UID and group in the namespace. -func (p *pvcBackupItemAction) findExistingVGS(backupUID types.UID, vgsLabelKey, group, namespace string) *VolumeGroupSnapshot { - var vgsList VolumeGroupSnapshotList - err := p.crClient.List( - context.TODO(), - &vgsList, - crclient.InNamespace(namespace), - crclient.MatchingLabels{ - "backup-uid": string(backupUID), - vgsLabelKey: group, - }, - ) - if err != nil { - p.log.Errorf("Error listing VGS: %v", err) - return nil - } - if len(vgsList.Items) > 0 { - return &vgsList.Items[0] - } - return nil -} - -// listGroupedPVCs returns all PVCs in the given namespace that have the specified VGS label key and group value. -func (p *pvcBackupItemAction) listGroupedPVCs(backup *velerov1api.Backup, namespace, vgsLabelKey, group string) ([]corev1api.PersistentVolumeClaim, error) { - var pvcList corev1api.PersistentVolumeClaimList - err := p.crClient.List(context.TODO(), &pvcList, crclient.InNamespace(namespace), crclient.MatchingLabels{vgsLabelKey: group}) - if err != nil { - return nil, err - } - return pvcList.Items, nil -} - -func (p *pvcBackupItemAction) createVolumeGroupSnapshot( - backup *velerov1api.Backup, - pvc corev1api.PersistentVolumeClaim, - pvcNames []string, - vgsLabelKey, group string, -) (*VolumeGroupSnapshot, error) { - // Generate a name for the new VGS object. - name := fmt.Sprintf("vgs-%s-%d", string(backup.UID), time.Now().Unix()) - - // Construct the VGS object - vgs := &VolumeGroupSnapshot{ - TypeMeta: metav1.TypeMeta{ - Kind: "VolumeGroupSnapshot", - APIVersion: "groupsnapshot.storage.k8s.io/v1beta1", - }, - ObjectMeta: metav1.ObjectMeta{ - Name: name, - Namespace: pvc.Namespace, - Annotations: map[string]string{ - "pvcList": strings.Join(pvcNames, ","), - }, - Labels: map[string]string{ - velerov1api.BackupUIDLabel: string(backup.UID), - vgsLabelKey: group, - }, - }, - Spec: VolumeGroupSnapshotSpec{ - // we will use default VGSClass for now - Source: VolumeGroupSnapshotSource{ - Selector: metav1.LabelSelector{ - MatchLabels: map[string]string{ - vgsLabelKey: group, - }, - }, - }, - }, - } - - // Create the VGS object - if err := p.crClient.Create(context.TODO(), vgs); err != nil { - return nil, err - } - return vgs, nil -} - -// extractPVCNames returns a slice of PVC names from the provided list. -func extractPVCNames(pvcs []corev1api.PersistentVolumeClaim) []string { - names := []string{} - for _, pvc := range pvcs { - names = append(names, pvc.Name) - } - return names -} - -``` - -- Add a new VolumeGroupSnapshot(VGS) BIA plugin -```go -// AppliesTo specifies that this plugin applies to VolumeGroupSnapshot objects. -func (p *vgsBackupItemAction) AppliesTo() (velero.ResourceSelector, error) { - return velero.ResourceSelector{ - IncludedResources: []string{"volumegroupsnapshots"}, - }, nil -} - -// Execute waits for all related VolumeSnapshot (VS) instances to be ready, then adds them -// along with the VolumeGroupSnapshotClass as additional backup items. -func (p *vgsBackupItemAction) Execute( +func (p *pvcBackupItemAction) Execute( item runtime.Unstructured, backup *velerov1api.Backup, ) ( @@ -454,55 +287,230 @@ func (p *vgsBackupItemAction) Execute( []velero.ResourceIdentifier, error, ) { - p.log.Infof("Starting VGS backup plugin Execute for item: %s", item.GetName()) + p.log.Info("Starting PVCBackupItemAction") - // Convert the unstructured object to a VolumeGroupSnapshot. - var vgs VolumeGroupSnapshot - if err := runtime.DefaultUnstructuredConverter.FromUnstructured(item.UnstructuredContent(), &vgs); err != nil { - return nil, nil, "", nil, errors.Wrap(err, "unable to convert item to VolumeGroupSnapshot") - } - - // Wait until all related VolumeSnapshots are ready. - p.log.Infof("Waiting for all related VolumeSnapshots for VGS %s to be ready", vgs.Name) - ready, err := waitForGroupedVolumeSnapshots(&vgs, vgs.Namespace, p.log) - if err != nil { - return nil, nil, "", nil, errors.Wrap(err, "error waiting for grouped VolumeSnapshots") - } - if !ready { - p.log.Infof("Not all related VolumeSnapshots are ready for VGS %s", vgs.Name) + if valid := p.validateBackup(*backup); !valid { return item, nil, "", nil, nil } - // Retrieve the related VolumeSnapshot instances. - groupedVS := getGroupedVolumeSnapshots(&vgs, vgs.Namespace) - var additionalItems []velero.ResourceIdentifier - for _, vs := range groupedVS { - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumesnapshots", - }, - Namespace: vs.Namespace, - Name: vs.Name, - }) + var pvc corev1api.PersistentVolumeClaim + if err := runtime.DefaultUnstructuredConverter.FromUnstructured( + item.UnstructuredContent(), &pvc, + ); err != nil { + return nil, nil, "", nil, errors.WithStack(err) } - // Add the VolumeGroupSnapshotClass as an additional item. - // Here we assume that the VGS spec defines the class name. - // We will be using the default VGSClass for now - vgsClassName := vgs.Spec.VolumeGroupSnapshotClassName - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumegroupsnapshotclasses", - }, - Namespace: "", - Name: vgsClassName, - }) + if valid, item, err := p.validatePVCandPV(pvc, item); !valid { + if err != nil { + return nil, nil, "", nil, err + } + return item, nil, "", nil, nil + } - p.log.Infof("VGS %s ready; adding %d additional items to backup", vgs.Name, len(additionalItems)) - return item, additionalItems, "", nil, nil + shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup( + item, + kuberesource.PersistentVolumeClaims, + *backup, + p.crClient, + p.log, + ) + if err != nil { + return nil, nil, "", nil, err + } + if !shouldSnapshot { + p.log.Debugf("CSI plugin skip snapshot for PVC %s according to VolumeHelper setting", pvc.Namespace+"/"+pvc.Name) + return nil, nil, "", nil, nil + } + + var additionalItems []velero.ResourceIdentifier + operationID := "" + var itemToUpdate []velero.ResourceIdentifier + + // vsRef will be used to apply common labels/annotations + var vsRef *corev1api.ObjectReference + + // VGS branch: check if PVC has the VGS label key set on it. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { + p.log.Infof("PVC %s has VGS label with group %s", pvc.Name, group) + // First, check if a VS created via a VGS workflow exists for this PVC. + existingVS, err := p.findExistingVSForBackup(backup.UID, backup.Name, pvc.Name, pvc.Namespace) + if err != nil { + return nil, nil, "", nil, err + } + if existingVS != nil && existingVS.Status.VolumeGroupSnapshotName != "" { + p.log.Infof("Existing VS %s found for PVC %s in group %s; skipping VGS creation", existingVS.Name, pvc.Name, group) + vsRef = &corev1api.ObjectReference{ + Namespace: existingVS.Namespace, + Name: existingVS.Name, + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumesnapshots", + }, + Namespace: existingVS.Namespace, + Name: existingVS.Name, + }) + } else { + // No existing VS found for the group; execute VGS creation workflow. + groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + pvcNames := extractPVCNames(groupedPVCs) + newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) + + // Wait for the VS objects created via VGS to have VolumeGroupSnapshotName in status. + if err := p.waitForVGSAssociatedVS(newVGS, pvc.Namespace, backup.Spec.CSISnapshotTimeout.Duration); err != nil { + return nil, nil, "", nil, err + } + // Update VS objects: remove VGS owner references and finalizers; add BackupName, BackupUUID and PVC name as labels. + if err := p.updateVGSCreatedVS(newVGS, backup); err != nil { + return nil, nil, "", nil, err + } + // Patch the VGSC deletionPolicy to Retain. + if err := p.patchVGSCDeletionPolicy(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + // Delete the VGS and VGSC objects to prevent cascading deletion. + if err := p.deleteVGSAndVGSC(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + + // Branch based on datamover flag. + if !boolptr.IsSetToTrue(backup.Spec.SnapshotMoveData) { + // Non-datamover: list VS objects created via VGS and use them. + vsList, err := p.listVSForVGSGroup(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + if len(vsList) == 0 { + return nil, nil, "", nil, errors.New("no VS objects found for VGS group " + group) + } + vsRef = &corev1api.ObjectReference{ + Namespace: vsList[0].Namespace, + Name: vsList[0].Name, + } + additionalItems = append(additionalItems, convertVSToResourceIdentifiers(vsList)...) + } else { + // Datamover: retrieve the VS for the PVC and create a DataUpload. + vs, err := p.getVSForPVC(backup, pvc, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + operationID = label.GetValidName(string(velerov1api.AsyncOperationIDPrefixDataUpload) + string(backup.UID) + "." + string(pvc.UID)) + dataUploadLog := p.log.WithFields(logrus.Fields{ + "Source PVC": fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name), + "VolumeSnapshot": fmt.Sprintf("%s/%s", vs.Namespace, vs.Name), + "Operation ID": operationID, + "Backup": backup.Name, + }) + // Wait until VS associated VSC snapshot handle is created. + _, err = csi.WaitUntilVSCHandleIsReady( + vs, + p.crClient, + p.log, + true, + backup.Spec.CSISnapshotTimeout.Duration, + ) + if err != nil { + dataUploadLog.Errorf("Fail to wait VolumeSnapshot turned to ReadyToUse: %s", err.Error()) + csi.CleanupVolumeSnapshot(vs, p.crClient, p.log) + return nil, nil, "", nil, errors.WithStack(err) + } + dataUploadLog.Info("Starting data upload of backup") + dataUpload, err := createDataUpload( + context.Background(), + backup, + p.crClient, + vs, + &pvc, + operationID, + ) + if err != nil { + dataUploadLog.WithError(err).Error("failed to submit DataUpload") + if deleteErr := p.crClient.Delete(context.TODO(), vs); deleteErr != nil { + if !apierrors.IsNotFound(deleteErr) { + dataUploadLog.WithError(deleteErr).Error("fail to delete VolumeSnapshot") + } + } + return item, nil, "", nil, nil + } else { + itemToUpdate = []velero.ResourceIdentifier{ + { + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }, + } + annotations[velerov1api.DataUploadNameAnnotation] = dataUpload.Namespace + "/" + dataUpload.Name + dataUploadLog.Info("DataUpload is submitted successfully.") + } + vsRef = &corev1api.ObjectReference{ + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }) + } + } + } else { + // Legacy workflow: PVC does not have a VGS label; create an individual VolumeSnapshot. + vs, err := p.createVolumeSnapshot(pvc, backup) + if err != nil { + return nil, nil, "", nil, err + } + vsRef = vs + additionalItems = []velero.ResourceIdentifier{ + { + GroupResource: kuberesource.VolumeSnapshots, + Namespace: vs.Namespace, + Name: vs.Name, + }, + } + } + + labels := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.BackupNameLabel: backup.Name, + } + + annotations := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.MustIncludeAdditionalItemAnnotation: "true", + } + + kubeutil.AddAnnotations(&pvc.ObjectMeta, annotations) + kubeutil.AddLabels(&pvc.ObjectMeta, labels) + + p.log.Infof("Returning from PVCBackupItemAction with %d additionalItems to backup", len(additionalItems)) + for _, ai := range additionalItems { + p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) + } + + pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&pvc) + if err != nil { + return nil, nil, "", nil, errors.WithStack(err) + } + + return &unstructured.Unstructured{Object: pvcMap}, + additionalItems, operationID, itemToUpdate, nil } + ``` ## Implementation From 2372c4ecf3a61e27799dbc7344252033c93e3b1b Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 8 Apr 2025 14:52:41 -0700 Subject: [PATCH 08/10] Update CSI plugin common branch flow and add mechanism to determine VGSClass Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 575 +++++++++++++++++--------------- 1 file changed, 301 insertions(+), 274 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 71e47d2ec..716a938b0 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -135,78 +135,101 @@ flowchart TD ``` #### Updates to CSI PVC plugin: -- When a PVC has a VGS label and no VS (created via VGS) exists: - - Create VGS: - - This triggers creation of the corresponding VGSC, VS, and VSC objects. - - Wait for VS Status: - - Wait until each VS (one per PVC in the group) has its volumeGroupSnapshotName set. This confirms that the snapshot controller has done its work. - - Update VS Objects: - - Remove owner references and VGS-related finalizers from the VS objects (decoupling them to prevent cascading deletion). - - Add backup metadata (BackupName, BackupUUID, PVC name) as labels. This metadata is later used to skip re-creating a VGS when another PVC of the same group is processed. - - Patch and Cleanup: - - Patch the VGSC deletionPolicy to Retain so that when you delete the VGSC, the underlying VSC (and the storage snapshots) remain. - - Delete the temporary VGS and VGSC objects. - - Branching: - - For non‑datamover cases, skip the creation of an individual VS (since it was created via VGS) and add the VS objects as additional items. - - For datamover cases, create DataUploads for the VS–PVC pair (using the VS created by the VGS workflow) and add those as additional items. - -- When a PVC has a VGS label and a VS created via an earlier VGS workflow already exists: - - List VS objects in the PVC’s namespace using labels (BackupUUID, BackupName, PVCName). - - Verify that a VS exists and that its status shows a non‑empty volumeGroupSnapshotName. - - If so, skip VGS (and VS) creation and continue with the legacy workflow. - - If a VS is found but it wasn’t created by the VGS workflow (i.e. it lacks the volumeGroupSnapshotName), then the backup for that PVC is failed, resulting in a partially failed backup. +The CSI PVC plugin now supports obtaining a VolumeSnapshot (VS) reference for a PVC in three ways, and then applies common branching for datamover and non‑datamover workflows: -- When a PVC does not have a VGS label: - - The legacy workflow is followed, creating an individual VolumeSnapshot as before. +- Scenario 1: PVC has a VGS label and no VS (created via the VGS workflow) exists for its volume group: + - Determine VGSClass: The plugin checks for CSI driver of all the grouped PVCs and then uses the corresponding VGSClass in VGS spec. If the grouped PVC has more than one CSI driver, VGS creation is skipped and backup fails. + - Create VGS: The plugin creates a new VolumeGroupSnapshot (VGS) for the PVC’s volume group. This action automatically triggers creation of the corresponding VGSC, VS, and VSC objects. + - Wait for VS Status: The plugin waits until each VS (one per PVC in the group) has its `volumeGroupSnapshotName` populated. This confirms that the snapshot controller has completed its work. `CSISnapshotTimeout` will be used here. + - Update VS Objects: Once the VS objects are provisioned, the plugin updates them by removing VGS owner references and VGS-related finalizers, and by adding backup metadata labels (including BackupName, BackupUUID, and PVC name). These labels are later used to detect an existing VS when processing another PVC of the same group. + - Patch and Cleanup: The plugin patches the deletionPolicy of the VGSC to "Retain" (ensuring that deletion of the VGSC does not remove the underlying VSC objects or storage snapshots) and then deletes the temporary VGS and VGSC objects. + +- Scenario 2: PVC has a VGS label and a VS created via an earlier VGS workflow already exists: + - The plugin lists VS objects in the PVC’s namespace using backup metadata labels (BackupUID, BackupName, and PVCName). + - It verifies that at least one VS has a non‑empty `volumeGroupSnapshotName` in its status. + - If such a VS exists, the plugin skips creating a new VGS (or VS) and proceeds with the legacy workflow using the existing VS. + - If a VS is found but its status does not indicate it was created by the VGS workflow (i.e. its `volumeGroupSnapshotName` is empty), the backup for that PVC is failed, resulting in a partially failed backup. +- Scenario 3: PVC does not have a VGS label: + - The legacy workflow is followed, and an individual VolumeSnapshot (VS) is created for the PVC. +- Common Branching for Datamover and Non‑datamover Workflows: + - Once a VS reference (`vsRef`) is determined—whether through the VGS workflow (Scenario 1 or 2) or the legacy workflow (Scenario 3)—the plugin then applies the common branching: + - Non‑datamover Case: The VS reference is directly added as an additional backup item. + + - Datamover Case: The plugin waits until the VS’s associated VSC snapshot handle is ready (using the configured CSISnapshotTimeout), then creates a DataUpload for the VS–PVC pair. The resulting DataUpload is then added as an additional backup item. ```mermaid flowchart TD -%% Section 1: Accept VGS Label from User + %% Section 1: Accept VGS Label from User subgraph Accept_Label - A1[User sets VGS label] - A2[User labels PVCs before backup] - A1 --> A2 + A1[User sets VGS label key using default velero.io/volume-group-snapshot or via server arg or Backup API spec] + A2[User labels PVCs before backup] + A1 --> A2 end -%% Section 2: PVC ItemBlockAction Plugin + %% Section 2: PVC ItemBlockAction Plugin Extension subgraph PVC_ItemBlockAction - B1[Check PVC is bound and has VolumeName] - B2[Add related PV to relatedItems] - B3[Add pods mounting PVC to relatedItems] - B4[Check if PVC has user-specified VGS label] - B5[List PVCs in namespace matching label criteria] - B6[Add matching PVCs to relatedItems] - B1 --> B2 --> B3 --> B4 - B4 -- Yes --> B5 - B5 --> B6 + B1[Check PVC is bound and has VolumeName] + B2[Add related PV to relatedItems] + B3[Add pods mounting PVC to relatedItems] + B4[Check if PVC has user-specified VGS label] + B5[List PVCs in namespace matching label criteria] + B6[Add matching PVCs to relatedItems] + B1 --> B2 --> B3 --> B4 + B4 -- Yes --> B5 + B5 --> B6 end -%% Section 3: CSI PVC Plugin Updates -subgraph CSI_PVC_Plugin -C1[For each PVC, check for VGS label] -C1 -- Yes --> C2[Case 1: VGS for volume group not yet created] -C2 -- True --> C3[Create new VGS triggering VGSC, VS and VSC creation] -C3 --> C4[Wait for VS objects to show volumeGroupSnapshotName using CSISnapshotTimeout] -C4 --> C5[Update VS objects: remove VGS owner refs and VGS finalizers; add BackupName, BackupUUID, PVC name as labels] -C5 --> C6[Patch VGSC deletionPolicy to Retain] -C6 --> C7[Delete VGS object] -C7 --> C8[Delete VGSC] -C8 --> C9[For non-datamover: Skip individual VS creation; add VS from VGS as additional item] -C8 --> C10[For datamover: Create DataUpload for VS-PVC pair; add DU as additional item] + %% Section 3: CSI PVC Plugin Updates + subgraph CSI_PVC_Plugin + C1[For each PVC, check for VGS label] + C1 -- Has VGS label --> C2[Determine scenario] + C1 -- No VGS label --> C16[Scenario 3: Legacy workflow - create individual VS] -C1 -- Yes --> C11[Case 2: VGS was created then deleted] -C11 --> C12[List VS using labels BackupUID, BackupName, PVCName] -C12 --> C13[Check if VS has non-empty volumeGroupSnapshotName] -C13 -- Yes --> C14[Skip VGS creation; use existing VS/DU; legacy workflow continues] -C13 -- No --> C15[Fail backup for PVC] + %% Scenario 1: No existing VS via VGS exists + subgraph Scenario1[Scenario 1: No existing VS via VGS] + S1[List grouped PVCs using VGS label] + S2[Determine CSI driver for grouped PVCs] + S3[If single CSI driver then select matching VGSClass; else fail backup] + S4[Create new VGS triggering VGSC, VS, and VSC creation] + S5[Wait for VS objects to have nonempty volumeGroupSnapshotName] + S6[Update VS objects; remove VGS owner refs and finalizers; add backup metadata labels] + S7[Patch VGSC deletionPolicy to Retain] + S8[Delete transient VGS and VGSC] + S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 + + end -C1 -- No --> C16[Case 3: PVC lacks VGS label; follow legacy workflow to create VS] -end + %% Scenario 2: Existing VS via VGS exists + subgraph Scenario2[Scenario 2: Existing VS via VGS exists] + S9[List VS objects using backup metadata - BackupUID, BackupName, PVCName] + S10[Check if any VS has nonempty volumeGroupSnapshotName] + S9 --> S10 + S10 -- Yes --> S11[Use existing VS] + S10 -- No --> S12[Fail backup for PVC] + end -%% Connect Main Sections -A2 --> B1 -B6 --> C1 + C2 -- Scenario1 applies --> S1 + C2 -- Scenario2 applies --> S9 + + %% Common Branch: After obtaining a VS reference + subgraph Common_Branch[Common Branch] + CB1[Obtain VS reference as vsRef] + CB2[If non-datamover, add vsRef as additional backup item] + CB3[If datamover, wait for VSC handle and create DataUpload; add DataUpload as additional backup item] + CB1 --> CB2 + CB1 --> CB3 + end + + %% Connect Scenario outcomes and legacy branch to the common branch + S8 --> CB1 + S11 --> CB1 + C16 --> CB1 + end + + %% Overall Flow Connections + A2 --> B1 + B6 --> C1 ``` @@ -278,239 +301,243 @@ Backup workflow: - Updates to [CSI PVC plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/backup/actions/csi/pvc_action.go#L200) (Update the Execute method): ```go func (p *pvcBackupItemAction) Execute( - item runtime.Unstructured, - backup *velerov1api.Backup, + item runtime.Unstructured, + backup *velerov1api.Backup, ) ( - runtime.Unstructured, - []velero.ResourceIdentifier, - string, - []velero.ResourceIdentifier, - error, + runtime.Unstructured, + []velero.ResourceIdentifier, + string, + []velero.ResourceIdentifier, + error, ) { - p.log.Info("Starting PVCBackupItemAction") + p.log.Info("Starting PVCBackupItemAction") - if valid := p.validateBackup(*backup); !valid { - return item, nil, "", nil, nil - } + // Validate backup policy and PVC/PV + if valid := p.validateBackup(*backup); !valid { + return item, nil, "", nil, nil + } - var pvc corev1api.PersistentVolumeClaim - if err := runtime.DefaultUnstructuredConverter.FromUnstructured( - item.UnstructuredContent(), &pvc, - ); err != nil { - return nil, nil, "", nil, errors.WithStack(err) - } + var pvc corev1api.PersistentVolumeClaim + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(item.UnstructuredContent(), &pvc); err != nil { + return nil, nil, "", nil, errors.WithStack(err) + } + if valid, item, err := p.validatePVCandPV(pvc, item); !valid { + if err != nil { + return nil, nil, "", nil, err + } + return item, nil, "", nil, nil + } - if valid, item, err := p.validatePVCandPV(pvc, item); !valid { - if err != nil { - return nil, nil, "", nil, err - } - return item, nil, "", nil, nil - } + shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup( + item, + kuberesource.PersistentVolumeClaims, + *backup, + p.crClient, + p.log, + ) + if err != nil { + return nil, nil, "", nil, err + } + if !shouldSnapshot { + p.log.Debugf("CSI plugin skip snapshot for PVC %s according to VolumeHelper setting", pvc.Namespace+"/"+pvc.Name) + return nil, nil, "", nil, nil + } - shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup( - item, - kuberesource.PersistentVolumeClaims, - *backup, - p.crClient, - p.log, - ) - if err != nil { - return nil, nil, "", nil, err - } - if !shouldSnapshot { - p.log.Debugf("CSI plugin skip snapshot for PVC %s according to VolumeHelper setting", pvc.Namespace+"/"+pvc.Name) - return nil, nil, "", nil, nil - } + var additionalItems []velero.ResourceIdentifier + var operationID string + var itemToUpdate []velero.ResourceIdentifier - var additionalItems []velero.ResourceIdentifier - operationID := "" - var itemToUpdate []velero.ResourceIdentifier + // vsRef will be our common reference to the VolumeSnapshot (VS) + var vsRef *corev1api.ObjectReference - // vsRef will be used to apply common labels/annotations - var vsRef *corev1api.ObjectReference + // Retrieve the VGS label key from the backup spec. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey - // VGS branch: check if PVC has the VGS label key set on it. - vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey - if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { - p.log.Infof("PVC %s has VGS label with group %s", pvc.Name, group) - // First, check if a VS created via a VGS workflow exists for this PVC. - existingVS, err := p.findExistingVSForBackup(backup.UID, backup.Name, pvc.Name, pvc.Namespace) - if err != nil { - return nil, nil, "", nil, err - } - if existingVS != nil && existingVS.Status.VolumeGroupSnapshotName != "" { - p.log.Infof("Existing VS %s found for PVC %s in group %s; skipping VGS creation", existingVS.Name, pvc.Name, group) - vsRef = &corev1api.ObjectReference{ - Namespace: existingVS.Namespace, - Name: existingVS.Name, - } - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumesnapshots", - }, - Namespace: existingVS.Namespace, - Name: existingVS.Name, - }) - } else { - // No existing VS found for the group; execute VGS creation workflow. - groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - pvcNames := extractPVCNames(groupedPVCs) - newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) - - // Wait for the VS objects created via VGS to have VolumeGroupSnapshotName in status. - if err := p.waitForVGSAssociatedVS(newVGS, pvc.Namespace, backup.Spec.CSISnapshotTimeout.Duration); err != nil { - return nil, nil, "", nil, err - } - // Update VS objects: remove VGS owner references and finalizers; add BackupName, BackupUUID and PVC name as labels. - if err := p.updateVGSCreatedVS(newVGS, backup); err != nil { - return nil, nil, "", nil, err - } - // Patch the VGSC deletionPolicy to Retain. - if err := p.patchVGSCDeletionPolicy(newVGS, pvc.Namespace); err != nil { - return nil, nil, "", nil, err - } - // Delete the VGS and VGSC objects to prevent cascading deletion. - if err := p.deleteVGSAndVGSC(newVGS, pvc.Namespace); err != nil { - return nil, nil, "", nil, err - } - - // Branch based on datamover flag. - if !boolptr.IsSetToTrue(backup.Spec.SnapshotMoveData) { - // Non-datamover: list VS objects created via VGS and use them. - vsList, err := p.listVSForVGSGroup(backup, pvc.Namespace, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - if len(vsList) == 0 { - return nil, nil, "", nil, errors.New("no VS objects found for VGS group " + group) - } - vsRef = &corev1api.ObjectReference{ - Namespace: vsList[0].Namespace, - Name: vsList[0].Name, - } - additionalItems = append(additionalItems, convertVSToResourceIdentifiers(vsList)...) - } else { - // Datamover: retrieve the VS for the PVC and create a DataUpload. - vs, err := p.getVSForPVC(backup, pvc, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - operationID = label.GetValidName(string(velerov1api.AsyncOperationIDPrefixDataUpload) + string(backup.UID) + "." + string(pvc.UID)) - dataUploadLog := p.log.WithFields(logrus.Fields{ - "Source PVC": fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name), - "VolumeSnapshot": fmt.Sprintf("%s/%s", vs.Namespace, vs.Name), - "Operation ID": operationID, - "Backup": backup.Name, - }) - // Wait until VS associated VSC snapshot handle is created. - _, err = csi.WaitUntilVSCHandleIsReady( - vs, - p.crClient, - p.log, - true, - backup.Spec.CSISnapshotTimeout.Duration, - ) - if err != nil { - dataUploadLog.Errorf("Fail to wait VolumeSnapshot turned to ReadyToUse: %s", err.Error()) - csi.CleanupVolumeSnapshot(vs, p.crClient, p.log) - return nil, nil, "", nil, errors.WithStack(err) - } - dataUploadLog.Info("Starting data upload of backup") - dataUpload, err := createDataUpload( - context.Background(), - backup, - p.crClient, - vs, - &pvc, - operationID, - ) - if err != nil { - dataUploadLog.WithError(err).Error("failed to submit DataUpload") - if deleteErr := p.crClient.Delete(context.TODO(), vs); deleteErr != nil { - if !apierrors.IsNotFound(deleteErr) { - dataUploadLog.WithError(deleteErr).Error("fail to delete VolumeSnapshot") - } - } - return item, nil, "", nil, nil - } else { - itemToUpdate = []velero.ResourceIdentifier{ - { - GroupResource: schema.GroupResource{ - Group: "velero.io", - Resource: "datauploads", - }, - Namespace: dataUpload.Namespace, - Name: dataUpload.Name, - }, - } - annotations[velerov1api.DataUploadNameAnnotation] = dataUpload.Namespace + "/" + dataUpload.Name - dataUploadLog.Info("DataUpload is submitted successfully.") - } - vsRef = &corev1api.ObjectReference{ - Namespace: dataUpload.Namespace, - Name: dataUpload.Name, - } - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "velero.io", - Resource: "datauploads", - }, - Namespace: dataUpload.Namespace, - Name: dataUpload.Name, - }) - } - } - } else { - // Legacy workflow: PVC does not have a VGS label; create an individual VolumeSnapshot. - vs, err := p.createVolumeSnapshot(pvc, backup) - if err != nil { - return nil, nil, "", nil, err - } - vsRef = vs - additionalItems = []velero.ResourceIdentifier{ - { - GroupResource: kuberesource.VolumeSnapshots, - Namespace: vs.Namespace, - Name: vs.Name, - }, - } - } + // Check if the PVC has the user-specified VGS label. + if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { + p.log.Infof("PVC %s has VGS label with group %s", pvc.Name, group) + // --- VGS branch --- + // 1. Check if a VS created via a VGS workflow exists for this PVC. + existingVS, err := p.findExistingVSForBackup(backup.UID, backup.Name, pvc.Name, pvc.Namespace) + if err != nil { + return nil, nil, "", nil, err + } + if existingVS != nil && existingVS.Status.VolumeGroupSnapshotName != "" { + p.log.Infof("Existing VS %s found for PVC %s in group %s; skipping VGS creation", existingVS.Name, pvc.Name, group) + vsRef = &corev1api.ObjectReference{ + Namespace: existingVS.Namespace, + Name: existingVS.Name, + } + } else { + // 2. No existing VS via VGS; execute VGS creation workflow. + groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + pvcNames := extractPVCNames(groupedPVCs) + // Determine the CSI driver used by the grouped PVCs. + driver, err := p.determineCSIDriver(groupedPVCs) + if err != nil { + return nil, nil, "", nil, errors.Wrap(err, "failed to determine CSI driver for grouped PVCs") + } + if driver == "" { + return nil, nil, "", nil, errors.New("multiple CSI drivers found for grouped PVCs; failing backup") + } + // Retrieve the appropriate VGSClass for the CSI driver. + vgsClass := p.getVGSClassForDriver(driver) + p.log.Infof("Determined CSI driver %s with VGSClass %s for PVC group %s", driver, vgsClass, group) - labels := map[string]string{ - velerov1api.VolumeSnapshotLabel: vsRef.Name, - velerov1api.BackupNameLabel: backup.Name, - } + newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group, vgsClass) + if err != nil { + return nil, nil, "", nil, err + } + p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) + + // Wait for the VS objects created via VGS to have volumeGroupSnapshotName in status. + if err := p.waitForVGSAssociatedVS(newVGS, pvc.Namespace, backup.Spec.CSISnapshotTimeout.Duration); err != nil { + return nil, nil, "", nil, err + } + // Update the VS objects: remove VGS owner references and finalizers; add backup metadata labels. + if err := p.updateVGSCreatedVS(newVGS, backup); err != nil { + return nil, nil, "", nil, err + } + // Patch the VGSC deletionPolicy to Retain. + if err := p.patchVGSCDeletionPolicy(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + // Delete the VGS and VGSC + if err := p.deleteVGSAndVGSC(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + // Fetch the VS that was created for this PVC via VGS. + vs, err := p.getVSForPVC(backup, pvc, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + vsRef = &corev1api.ObjectReference{ + Namespace: vs.Namespace, + Name: vs.Name, + } + } + } else { + // Legacy workflow: PVC does not have a VGS label; create an individual VS. + vs, err := p.createVolumeSnapshot(pvc, backup) + if err != nil { + return nil, nil, "", nil, err + } + vsRef = &corev1api.ObjectReference{ + Namespace: vs.Namespace, + Name: vs.Name, + } + } - annotations := map[string]string{ - velerov1api.VolumeSnapshotLabel: vsRef.Name, - velerov1api.MustIncludeAdditionalItemAnnotation: "true", - } + // --- Common Branch --- + // Now we have vsRef populated from one of the above cases. + // Branch further based on backup.Spec.SnapshotMoveData. + if boolptr.IsSetToTrue(backup.Spec.SnapshotMoveData) { + // Datamover case: + operationID = label.GetValidName( + string(velerov1api.AsyncOperationIDPrefixDataUpload) + string(backup.UID) + "." + string(pvc.UID), + ) + dataUploadLog := p.log.WithFields(logrus.Fields{ + "Source PVC": fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name), + "VolumeSnapshot": fmt.Sprintf("%s/%s", vsRef.Namespace, vsRef.Name), + "Operation ID": operationID, + "Backup": backup.Name, + }) + // Retrieve the current VS using vsRef + vs := &snapshotv1api.VolumeSnapshot{} + if err := p.crClient.Get(context.TODO(), crclient.ObjectKey{Namespace: vsRef.Namespace, Name: vsRef.Name}, vs); err != nil { + return nil, nil, "", nil, errors.Wrapf(err, "failed to get VolumeSnapshot %s", vsRef.Name) + } + // Wait until the VS-associated VSC snapshot handle is ready. + _, err := csi.WaitUntilVSCHandleIsReady( + vs, + p.crClient, + p.log, + true, + backup.Spec.CSISnapshotTimeout.Duration, + ) + if err != nil { + dataUploadLog.Errorf("Failed to wait for VolumeSnapshot to become ReadyToUse: %s", err.Error()) + csi.CleanupVolumeSnapshot(vs, p.crClient, p.log) + return nil, nil, "", nil, errors.WithStack(err) + } + dataUploadLog.Info("Starting data upload of backup") + dataUpload, err := createDataUpload( + context.Background(), + backup, + p.crClient, + vs, + &pvc, + operationID, + ) + if err != nil { + dataUploadLog.WithError(err).Error("Failed to submit DataUpload") + if deleteErr := p.crClient.Delete(context.TODO(), vs); deleteErr != nil && !apierrors.IsNotFound(deleteErr) { + dataUploadLog.WithError(deleteErr).Error("Failed to delete VolumeSnapshot") + } + return item, nil, "", nil, nil + } + dataUploadLog.Info("DataUpload submitted successfully") + itemToUpdate = []velero.ResourceIdentifier{ + { + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }, + } + annotations[velerov1api.DataUploadNameAnnotation] = dataUpload.Namespace + "/" + dataUpload.Name + // For the datamover case, add the dataUpload as an additional item directly. + vsRef = &corev1api.ObjectReference{ + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }) + } else { + // Non-datamover case: + // Use vsRef for snapshot purposes. + additionalItems = append(additionalItems, convertVSToResourceIdentifiersFromRef(vsRef)...) + p.log.Infof("VolumeSnapshot additional item added for VS %s", vsRef.Name) + } - kubeutil.AddAnnotations(&pvc.ObjectMeta, annotations) - kubeutil.AddLabels(&pvc.ObjectMeta, labels) + // Update PVC metadata with common labels and annotations. + labels := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.BackupNameLabel: backup.Name, + } + annotations := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.MustIncludeAdditionalItemAnnotation: "true", + } + kubeutil.AddAnnotations(&pvc.ObjectMeta, annotations) + kubeutil.AddLabels(&pvc.ObjectMeta, labels) - p.log.Infof("Returning from PVCBackupItemAction with %d additionalItems to backup", len(additionalItems)) - for _, ai := range additionalItems { - p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) - } + p.log.Infof("Returning from PVCBackupItemAction with %d additionalItems to backup", len(additionalItems)) + for _, ai := range additionalItems { + p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) + } - pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&pvc) - if err != nil { - return nil, nil, "", nil, errors.WithStack(err) - } + pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&pvc) + if err != nil { + return nil, nil, "", nil, errors.WithStack(err) + } - return &unstructured.Unstructured{Object: pvcMap}, - additionalItems, operationID, itemToUpdate, nil + return &unstructured.Unstructured{Object: pvcMap}, + additionalItems, operationID, itemToUpdate, nil } + ``` ## Implementation From 71b889aa6efc9a569f0ca888794bca4f04e003fc Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 22 Apr 2025 13:16:58 -0700 Subject: [PATCH 09/10] Update VGSClass determination mechanism Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 716a938b0..acc13455d 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -138,7 +138,30 @@ flowchart TD The CSI PVC plugin now supports obtaining a VolumeSnapshot (VS) reference for a PVC in three ways, and then applies common branching for datamover and non‑datamover workflows: - Scenario 1: PVC has a VGS label and no VS (created via the VGS workflow) exists for its volume group: - - Determine VGSClass: The plugin checks for CSI driver of all the grouped PVCs and then uses the corresponding VGSClass in VGS spec. If the grouped PVC has more than one CSI driver, VGS creation is skipped and backup fails. + - Determine VGSClass: The plugin will pick `VolumeGroupSnapshotClass` by following the same tier based precedence as it does for individual `VolumeSnapshotClasses`: + - Default by Label: Use the one VGSClass labeled + ```yaml + metadata: + labels: + velero.io/csi-volumegroupsnapshot-class: "true" + + ``` + whose `spec.driver` matches the CSI driver used by the PVCs. + - Backup‑level Override: If the Backup CR has an annotation + ```yaml + metadata: + annotations: + velero.io/csi-volumegroupsnapshot-class_: + ``` + (with equal to the PVCs’ CSI driver), use that class. + - PVC‑level Override: Finally, if the PVC itself carries an annotation + ```yaml + metadata: + annotations: + velero.io/csi-volume-group-snapshot-class: + ``` + and that class exists, use it. + At each step, if the plugin finds zero or multiple matching classes, VGS creation is skipped and backup fails. - Create VGS: The plugin creates a new VolumeGroupSnapshot (VGS) for the PVC’s volume group. This action automatically triggers creation of the corresponding VGSC, VS, and VSC objects. - Wait for VS Status: The plugin waits until each VS (one per PVC in the group) has its `volumeGroupSnapshotName` populated. This confirms that the snapshot controller has completed its work. `CSISnapshotTimeout` will be used here. - Update VS Objects: Once the VS objects are provisioned, the plugin updates them by removing VGS owner references and VGS-related finalizers, and by adding backup metadata labels (including BackupName, BackupUUID, and PVC name). These labels are later used to detect an existing VS when processing another PVC of the same group. From b30e43998aae479481d0b19b0b81ace94713748e Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Wed, 23 Apr 2025 19:52:22 -0700 Subject: [PATCH 10/10] Add notes regarding compat, perf, reqs and testing Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 50 +++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index acc13455d..83bc5a574 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -2,6 +2,16 @@ This proposal outlines the design and implementation plan for incorporating VolumeGroupSnapshot support into Velero. The enhancement will allow Velero to perform consistent, atomic snapshots of groups of Volumes using the new Kubernetes [VolumeGroupSnapshot API](https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/). This capability is especially critical for stateful applications that rely on multiple volumes to ensure data consistency, such as databases and analytics workloads. +## Glossary & Abbreviation + +Terminology used in this document: +- VGS: VolumeGroupSnapshot +- VS: VolumeSnapshot +- VGSC: VolumeGroupSnapshotContent +- VSC: VolumeSnapshotContent +- VGSClass: VolumeGroupSnapshotClass +- VSClass: VolumeSnapshotClass + ## Background Velero currently enables snapshot-based backups on an individual Volume basis through CSI drivers. However, modern stateful applications often require multiple volumes for data, logs, and backups. This distributed data architecture increases the risk of inconsistencies when volumes are captured individually. Kubernetes has introduced the VolumeGroupSnapshot(VGS) API [(KEP-3476)](https://github.com/kubernetes/enhancements/pull/1551), which allows for the atomic snapshotting of multiple volumes in a coordinated manner. By integrating this feature, Velero can offer enhanced disaster recovery for multi-volume applications, ensuring consistency across all related data. @@ -20,16 +30,23 @@ Velero currently enables snapshot-based backups on an individual Volume basis th ### Backup workflow: #### Accept the label to be used for VGS from the user: - Accept the label from the user, we will do this in 3 ways: - - Firstly, we will have a hard-coded default label key like `velero.io/volume-group-snapshot` that the users can directly use on their PVCs. - - Secondly, we will let the users override this default VGS label via a velero server arg, `--volume-group-nsaphot-label-key`, if needed. - - And Finally we will have the option to override the default label via Backup API spec, `backup.spec.volumeGroupSnapshotLabelKey` - - In all the instances, the VGS label key will be present on the backup spec, this makes the label key accessible to plugins during the execution of backup operation. + - Firstly, we will have a hard-coded default label key like `velero.io/volume-group-snapshot` that the users can directly use on their PVCs. + - Secondly, we will let the users override this default VGS label via a velero server arg, `--volume-group-nsaphot-label-key`, if needed. + - And Finally we will have the option to override the default label via Backup API spec, `backup.spec.volumeGroupSnapshotLabelKey` + - In all the instances, the VGS label key will be present on the backup spec, this makes the label key accessible to plugins during the execution of backup operation. - This label will enable velero to filter the PVC to be included in the VGS spec. - Users will have to label the PVCs before invoking the backup operation. - This label would act as a group identifier for the PVCs to be grouped under a specific VGS. - It will be used to collect the PVCs to be used for a particular instance of VGS object. -**Note:** Modifying or adding VGS label on PVCs during an active backup operation may lead to unexpected or undesirable backup results. To avoid inconsistencies, ensure PVC labels remain unchanged throughout the backup execution. +**Note:** + - Modifying or adding VGS label on PVCs during an active backup operation may lead to unexpected or undesirable backup results. To avoid inconsistencies, ensure PVC labels remain unchanged throughout the backup execution. + - Label Key Precedence: When determining which label key to use for grouping PVCs into a VolumeGroupSnapshot, Velero applies overrides in the following order (highest to lowest): + - Backup API spec (`backup.spec.volumeGroupSnapshotLabelKey`) + - Server flag (`--volume-group-snapshot-label-key`) + - Built-in default (`velero.io/volume-group-snapshot`) + + Whichever key wins this precedence is then injected into the Backup spec so that all Velero plugins can uniformly discover and use it during the backup execution. #### Changes to the Existing PVC ItemBlockAction plugin: - Currently the PVC IBA plugin is applied to PVCs and adds the RelatedItems for the particular PVC into the ItemBlock. - At first it checks whether the PVC is bound and VolumeName is non-empty. @@ -569,3 +586,26 @@ This design proposal is targeted for velero 1.16. The implementation of this proposed design is targeted for velero 1.17. +**Note:** +- VGS support isn't a requirement on restore. The design does not have any VGS related elements/considerations in the restore workflow. + +## Requirements and Assumptions +- Kubernetes Version: + - Minimum: v1.32.0 or later, since the VolumeGroupSnapshot API goes beta in 1.32. + - Assumption: CRDs for `VolumeGroupSnapshot`, `VolumeGroupSnapshotClass`, and `VolumeGroupSnapshotContent` are already installed. + +- VolumeGroupSnapshot API Availability: + - If the VGS API group (`groupsnapshot.storage.k8s.io/v1beta1`) is not present, Velero backup will fail. + +- CSI Driver Compatibility + - Only CSI drivers that implement the VolumeGroupSnapshot admission and controller support this feature. + - Upon VGS creation, we assume the driver will atomically snapshot all matching PVCs; if it does not, the plugin may time out. + +## Performance Considerations +- Use VGS if you have many similar volumes that must be snapped together and you want to minimize API/server load. +- Use individual VS if you have only a few volumes, or want one‐volume failures to be isolated. + +## Testing Strategy + +- Unit tests: We will add targeted unit tests to cover all new code paths—including existing-VS detection, VGS creation, legacy VS fallback, and error scenarios. +- E2E tests: For E2E we would need, a Kind cluster with a CSI driver that supports group snapshots, deploy an application with multiple PVCs, execute a Velero backup and restore, and verify that VGS is created, all underlying VS objects reach ReadyToUse, and every PVC is restored successfully. \ No newline at end of file