vmware-tanzu
/
velero
mirror of https://github.com/vmware-tanzu/velero.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #8684 from blackpiglet/7979_fix 

7979 fix
pull/3355/merge
Xun Jiang/Bruce Jiang 2025-02-25 13:27:01 +08:00 committed by GitHub
parent ad08c7a3ff 6b7dd12bf7
commit f79b825cf1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
25 changed files with 589 additions and 786 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
changelogs/unreleased/8684-blackpiglet Normal file
View File

@ -0,0 +1 @@
Clean artifacts generated during CSI B/R.

181
design/clean_artifacts_in_csi_flow.md
View File

@ -95,7 +95,7 @@ func DeleteReadyVolumeSnapshot(
### Restore ### Restore
#### Restore the VolumeSnapshotContent from the backup instead of creating a new one dynamically #### Restore the VolumeSnapshotContent
The current behavior of VSC restoration is that the VSC from the backup is restore, and the restored VS also triggers creating a new VSC dynamically. The current behavior of VSC restoration is that the VSC from the backup is restore, and the restored VS also triggers creating a new VSC dynamically.
Two VSCs created for the same VS in one restore seems not right. Two VSCs created for the same VS in one restore seems not right.
@ -106,6 +106,16 @@ If the `SkipRestore` is set true in the restore action's result, the secret retu
As a result, restore the VSC from the backup, and setup the VSC and the VS's relation is a better choice. As a result, restore the VSC from the backup, and setup the VSC and the VS's relation is a better choice.
Another consideration is the VSC name should not be the same as the backed-up VSC's, because the older version Velero's restore and backup keep the VSC after completion.
There's high possibility that the restore will fail due to the VSC already exists in the cluster.
Multiple restores of the same backup will also meet the same problem.
The proposed solution is using the restore's UID and the VS's name to generate sha256 hash value as the new VSC name. Both the VS and VSC RestoreItemAction can access those UIDs, and it will avoid the conflicts issues.
The restored VS name also shares the same generated name.
The VS-referenced VSC name and the VSC's snapshot handle name are in their status. The VS-referenced VSC name and the VSC's snapshot handle name are in their status.
Velero restore process purges the restore resources' metadata and status before running the RestoreItemActions. Velero restore process purges the restore resources' metadata and status before running the RestoreItemActions.
@ -114,6 +124,78 @@ As a result, we cannot read these information in the VS and VSC RestoreItemActio
Fortunately, RestoreItemAction input parameters includes the `ItemFromBackup`. The status is intact in `ItemFromBackup`. Fortunately, RestoreItemAction input parameters includes the `ItemFromBackup`. The status is intact in `ItemFromBackup`.
``` go
func (p *volumeSnapshotRestoreItemAction) Execute(
input *velero.RestoreItemActionExecuteInput,
) (*velero.RestoreItemActionExecuteOutput, error) {
p.log.Info("Starting VolumeSnapshotRestoreItemAction")
if boolptr.IsSetToFalse(input.Restore.Spec.RestorePVs) {
p.log.Infof("Restore %s/%s did not request for PVs to be restored.",
input.Restore.Namespace, input.Restore.Name)
return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil
}
var vs snapshotv1api.VolumeSnapshot
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
input.Item.UnstructuredContent(), &vs); err != nil {
return &velero.RestoreItemActionExecuteOutput{},
errors.Wrapf(err, "failed to convert input.Item from unstructured")
}
var vsFromBackup snapshotv1api.VolumeSnapshot
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
input.ItemFromBackup.UnstructuredContent(), &vsFromBackup); err != nil {
return &velero.RestoreItemActionExecuteOutput{},
errors.Wrapf(err, "failed to convert input.Item from unstructured")
}
// If cross-namespace restore is configured, change the namespace
// for VolumeSnapshot object to be restored
newNamespace, ok := input.Restore.Spec.NamespaceMapping[vs.GetNamespace()]
if !ok {
// Use original namespace
newNamespace = vs.Namespace
}
if csiutil.IsVolumeSnapshotExists(newNamespace, vs.Name, p.crClient) {
p.log.Debugf("VolumeSnapshot %s already exists in the cluster. Return without change.", vs.Namespace+"/"+vs.Name)
return &velero.RestoreItemActionExecuteOutput{UpdatedItem: input.Item}, nil
}
newVSCName := generateSha256FromRestoreAndVsUID(string(input.Restore.UID), string(vsFromBackup.UID))
// Reset Spec to convert the VolumeSnapshot from using
// the dynamic VolumeSnapshotContent to the static one.
resetVolumeSnapshotSpecForRestore(&vs, &newVSCName)
// Reset VolumeSnapshot annotation. By now, only change
// DeletionPolicy to Retain.
resetVolumeSnapshotAnnotation(&vs)
vsMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&vs)
if err != nil {
p.log.Errorf("Fail to convert VS %s to unstructured", vs.Namespace+"/"+vs.Name)
return nil, errors.WithStack(err)
}
p.log.Infof(`Returning from VolumeSnapshotRestoreItemAction with
no additionalItems`)
return &velero.RestoreItemActionExecuteOutput{
UpdatedItem: &unstructured.Unstructured{Object: vsMap},
AdditionalItems: []velero.ResourceIdentifier{},
}, nil
}
// generateSha256FromRestoreAndVsUID Use the restore UID and the VS UID to generate the new VSC name.
// By this way, VS and VSC RIA action can get the same VSC name.
func generateSha256FromRestoreAndVsUID(restoreUID string, vsUID string) string {
sha256Bytes := sha256.Sum256([]byte(restoreUID + "/" + vsUID))
return "vsc-" + hex.EncodeToString(sha256Bytes[:])
}
```
#### Restore the VolumeSnapshot
``` go ``` go
// Execute restores a VolumeSnapshotContent object without modification // Execute restores a VolumeSnapshotContent object without modification
// returning the snapshot lister secret, if any, as additional items to restore. // returning the snapshot lister secret, if any, as additional items to restore.
@ -128,9 +210,9 @@ func (p *volumeSnapshotContentRestoreItemAction) Execute(
p.log.Info("Starting VolumeSnapshotContentRestoreItemAction") p.log.Info("Starting VolumeSnapshotContentRestoreItemAction")
var snapCont snapshotv1api.VolumeSnapshotContent var vsc snapshotv1api.VolumeSnapshotContent
if err := runtime.DefaultUnstructuredConverter.FromUnstructured( if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
input.Item.UnstructuredContent(), &snapCont); err != nil { input.Item.UnstructuredContent(), &vsc); err != nil {
return &velero.RestoreItemActionExecuteOutput{}, return &velero.RestoreItemActionExecuteOutput{},
errors.Wrapf(err, "failed to convert input.Item from unstructured") errors.Wrapf(err, "failed to convert input.Item from unstructured")
} }
@ -144,39 +226,42 @@ func (p *volumeSnapshotContentRestoreItemAction) Execute(
// If cross-namespace restore is configured, change the namespace // If cross-namespace restore is configured, change the namespace
// for VolumeSnapshot object to be restored // for VolumeSnapshot object to be restored
newNamespace, ok := input.Restore.Spec.NamespaceMapping[snapCont.Spec.VolumeSnapshotRef.Namespace] newNamespace, ok := input.Restore.Spec.NamespaceMapping[vsc.Spec.VolumeSnapshotRef.Namespace]
if ok { if ok {
// Update the referenced VS namespace to the mapping one. // Update the referenced VS namespace to the mapping one.
snapCont.Spec.VolumeSnapshotRef.Namespace = newNamespace vsc.Spec.VolumeSnapshotRef.Namespace = newNamespace
} }
// Reset VSC name to align with VS.
vsc.Name = generateSha256FromRestoreAndVsUID(string(input.Restore.UID), string(vscFromBackup.Spec.VolumeSnapshotRef.UID))
// Reset the ResourceVersion and UID of referenced VolumeSnapshot. // Reset the ResourceVersion and UID of referenced VolumeSnapshot.
snapCont.Spec.VolumeSnapshotRef.ResourceVersion = "" vsc.Spec.VolumeSnapshotRef.ResourceVersion = ""
snapCont.Spec.VolumeSnapshotRef.UID = "" vsc.Spec.VolumeSnapshotRef.UID = ""
// Set the DeletionPolicy to Retain to avoid VS deletion will not trigger snapshot deletion // Set the DeletionPolicy to Retain to avoid VS deletion will not trigger snapshot deletion
snapCont.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentRetain vsc.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentRetain
if vscFromBackup.Status != nil && vscFromBackup.Status.SnapshotHandle != nil { if vscFromBackup.Status != nil && vscFromBackup.Status.SnapshotHandle != nil {
snapCont.Spec.Source.VolumeHandle = nil vsc.Spec.Source.VolumeHandle = nil
snapCont.Spec.Source.SnapshotHandle = vscFromBackup.Status.SnapshotHandle vsc.Spec.Source.SnapshotHandle = vscFromBackup.Status.SnapshotHandle
} else { } else {
p.log.Errorf("fail to get snapshot handle from VSC %s status", snapCont.Name) p.log.Errorf("fail to get snapshot handle from VSC %s status", vsc.Name)
return nil, errors.Errorf("fail to get snapshot handle from VSC %s status", snapCont.Name) return nil, errors.Errorf("fail to get snapshot handle from VSC %s status", vsc.Name)
} }
additionalItems := []velero.ResourceIdentifier{} additionalItems := []velero.ResourceIdentifier{}
if csi.IsVolumeSnapshotContentHasDeleteSecret(&snapCont) { if csi.IsVolumeSnapshotContentHasDeleteSecret(&vsc) {
additionalItems = append(additionalItems, additionalItems = append(additionalItems,
velero.ResourceIdentifier{ velero.ResourceIdentifier{
GroupResource: schema.GroupResource{Group: "", Resource: "secrets"}, GroupResource: schema.GroupResource{Group: "", Resource: "secrets"},
Name: snapCont.Annotations[velerov1api.PrefixedSecretNameAnnotation], Name: vsc.Annotations[velerov1api.PrefixedSecretNameAnnotation],
Namespace: snapCont.Annotations[velerov1api.PrefixedSecretNamespaceAnnotation], Namespace: vsc.Annotations[velerov1api.PrefixedSecretNamespaceAnnotation],
}, },
) )
} }
vscMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&snapCont) vscMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&vsc)
if err != nil { if err != nil {
return nil, errors.WithStack(err) return nil, errors.WithStack(err)
} }
@ -190,61 +275,6 @@ func (p *volumeSnapshotContentRestoreItemAction) Execute(
} }
``` ```
Because VSC is not restored dynamically, if we run restores two times for the same backup in the same cluster, the second restore will fail due to the VSC is there in the cluster, and it is already bound to an existing VS.
To avoid this issue, it's better to delete the restored VS and VSC after restore completes.
The VolumeSnapshot and VolumeSnapshotContent are delete in the `restore_finalizer_controller`.
``` go
func (ctx *finalizerContext) deleteVolumeSnapshotAndVolumeSnapshotContent() (errs results.Result) {
for _, operation := range ctx.restoreItemOperationList.items {
if operation.Spec.RestoreItemAction == constant.PluginCsiVolumeSnapshotRestoreRIA &&
operation.Status.Phase == itemoperation.OperationPhaseCompleted {
if operation.Spec.OperationID == "" || !strings.Contains(operation.Spec.OperationID, "/") {
ctx.logger.Errorf("invalid OperationID: %s", operation.Spec.OperationID)
errs.Add("", errors.Errorf("invalid OperationID: %s", operation.Spec.OperationID))
continue
}
operationIDParts := strings.Split(operation.Spec.OperationID, "/")
vs := new(snapshotv1api.VolumeSnapshot)
vsc := new(snapshotv1api.VolumeSnapshotContent)
if err := ctx.crClient.Get(
context.TODO(),
client.ObjectKey{Namespace: operationIDParts[0], Name: operationIDParts[1]},
vs,
); err != nil {
ctx.logger.Errorf("Fail to get the VolumeSnapshot %s: %s", operation.Spec.OperationID, err.Error())
errs.Add(operationIDParts[0], errors.Errorf("Fail to get the VolumeSnapshot %s: %s", operation.Spec.OperationID, err.Error()))
continue
}
if err := ctx.crClient.Delete(context.TODO(), vs); err != nil {
ctx.logger.Errorf("Fail to delete VolumeSnapshot %s: %s", operation.Spec.OperationID, err.Error())
errs.Add(vs.Namespace, err)
}
if vs.Status != nil && vs.Status.BoundVolumeSnapshotContentName != nil {
vsc.Name = *vs.Status.BoundVolumeSnapshotContentName
} else {
ctx.logger.Errorf("VolumeSnapshotContent %s is not ready.", vsc.Name)
errs.Add("", errors.Errorf("VolumeSnapshotContent %s is not ready.", vsc.Name))
continue
}
if err := ctx.crClient.Delete(context.TODO(), vsc); err != nil {
ctx.logger.Errorf("Fail to delete VolumeSnapshotContent %s: %s", vsc.Name, err.Error())
errs.Add("", errors.Errorf("Fail to delete the VolumeSnapshotContent %s", err))
}
}
}
return errs
}
```
### Backup Sync ### Backup Sync
csi-volumesnapshotclasses.json, csi-volumesnapshotcontents.json, and csi-volumesnapshots.json are CSI-related metadata files in the BSL for each backup. csi-volumesnapshotclasses.json, csi-volumesnapshotcontents.json, and csi-volumesnapshots.json are CSI-related metadata files in the BSL for each backup.
@ -266,8 +296,17 @@ For the VSC DeleteItemAction, we need to generate a VSC. Because we only care ab
Create a static VSC, then point it to a pseudo VS, and reference to the snapshot handle should be enough. Create a static VSC, then point it to a pseudo VS, and reference to the snapshot handle should be enough.
To avoid the created VSC conflict with older version Velero B/R generated ones, the VSC name is set to `vsc-uuid`.
The following is an example of the implementation. The following is an example of the implementation.
``` go ``` go
uuid, err := uuid.NewRandom()
if err != nil {
p.log.WithError(err).Errorf("Fail to generate the UUID to create VSC %s", snapCont.Name)
return errors.Wrapf(err, "Fail to generate the UUID to create VSC %s", snapCont.Name)
}
snapCont.Name = "vsc-" + uuid.String()
snapCont.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentDelete snapCont.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentDelete
snapCont.Spec.Source = snapshotv1api.VolumeSnapshotContentSource{ snapCont.Spec.Source = snapshotv1api.VolumeSnapshotContentSource{

120
internal/delete/actions/csi/volumesnapshot_action.go
View File

@ -1,120 +0,0 @@
/*
Copyright the Velero contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package csi
import (
"context"
"fmt"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
crclient "sigs.k8s.io/controller-runtime/pkg/client"
"github.com/vmware-tanzu/velero/pkg/client"
plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common"
"github.com/vmware-tanzu/velero/pkg/plugin/velero"
"github.com/vmware-tanzu/velero/pkg/util/csi"
kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube"
)
// volumeSnapshotDeleteItemAction is a backup item action plugin for Velero.
type volumeSnapshotDeleteItemAction struct {
log logrus.FieldLogger
crClient crclient.Client
}
// AppliesTo returns information indicating that the
// VolumeSnapshotBackupItemAction should be invoked to backup
// VolumeSnapshots.
func (p *volumeSnapshotDeleteItemAction) AppliesTo() (velero.ResourceSelector, error) {
p.log.Debug("VolumeSnapshotBackupItemAction AppliesTo")
return velero.ResourceSelector{
IncludedResources: []string{"volumesnapshots.snapshot.storage.k8s.io"},
}, nil
}
func (p *volumeSnapshotDeleteItemAction) Execute(
input *velero.DeleteItemActionExecuteInput,
) error {
p.log.Info("Starting VolumeSnapshotDeleteItemAction for volumeSnapshot")
var vs snapshotv1api.VolumeSnapshot
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
input.Item.UnstructuredContent(),
&vs,
); err != nil {
return errors.Wrapf(err, "failed to convert input.Item from unstructured")
}
// We don't want this DeleteItemAction plugin to delete VolumeSnapshot
// taken outside of Velero. So skip deleting VolumeSnapshot objects
// that were not created in the process of creating the Velero
// backup being deleted.
if !kubeutil.HasBackupLabel(&vs.ObjectMeta, input.Backup.Name) {
p.log.Info(
"VolumeSnapshot %s/%s was not taken by backup %s, skipping deletion",
vs.Namespace, vs.Name, input.Backup.Name,
)
return nil
}
p.log.Infof("Deleting VolumeSnapshot %s/%s", vs.Namespace, vs.Name)
if vs.Status != nil && vs.Status.BoundVolumeSnapshotContentName != nil {
// we patch the DeletionPolicy of the VolumeSnapshotContent
// to set it to Delete. This ensures that the volume snapshot
// in the storage provider is also deleted.
err := csi.SetVolumeSnapshotContentDeletionPolicy(
*vs.Status.BoundVolumeSnapshotContentName,
p.crClient,
)
if err != nil && !apierrors.IsNotFound(err) {
return errors.Wrapf(
err,
fmt.Sprintf("failed to patch DeletionPolicy of volume snapshot %s/%s",
vs.Namespace, vs.Name),
)
}
if apierrors.IsNotFound(err) {
return nil
}
}
err := p.crClient.Delete(context.TODO(), &vs)
if err != nil && !apierrors.IsNotFound(err) {
return err
}
return nil
}
func NewVolumeSnapshotDeleteItemAction(f client.Factory) plugincommon.HandlerInitializer {
return func(logger logrus.FieldLogger) (any, error) {
crClient, err := f.KubebuilderClient()
if err != nil {
return nil, errors.WithStack(err)
}
return &volumeSnapshotDeleteItemAction{
log: logger,
crClient: crClient,
}, nil
}
}

151
internal/delete/actions/csi/volumesnapshot_action_test.go
View File

@ -1,151 +0,0 @@
/*
Copyright the Velero contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package csi
import (
"context"
"fmt"
"testing"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/require"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/builder"
factorymocks "github.com/vmware-tanzu/velero/pkg/client/mocks"
"github.com/vmware-tanzu/velero/pkg/plugin/velero"
velerotest "github.com/vmware-tanzu/velero/pkg/test"
)
func TestVSExecute(t *testing.T) {
tests := []struct {
name string
item runtime.Unstructured
vs *snapshotv1api.VolumeSnapshot
backup *velerov1api.Backup
createVS bool
expectErr bool
}{
{
name: "VolumeSnapshot doesn't have backup label",
item: velerotest.UnstructuredOrDie(
`
{
"apiVersion": "snapshot.storage.k8s.io/v1",
"kind": "VolumeSnapshot",
"metadata": {
"namespace": "ns",
"name": "foo"
}
}
`,
),
backup: builder.ForBackup("velero", "backup").Result(),
expectErr: false,
},
{
name: "VolumeSnapshot doesn't exist in the cluster",
vs: builder.ForVolumeSnapshot("foo", "bar").
ObjectMeta(builder.WithLabelsMap(
map[string]string{velerov1api.BackupNameLabel: "backup"},
)).Status().
BoundVolumeSnapshotContentName("vsc").
Result(),
backup: builder.ForBackup("velero", "backup").Result(),
expectErr: true,
},
{
name: "Normal case, VolumeSnapshot should be deleted",
vs: builder.ForVolumeSnapshot("foo", "bar").
ObjectMeta(builder.WithLabelsMap(
map[string]string{velerov1api.BackupNameLabel: "backup"},
)).Status().
BoundVolumeSnapshotContentName("vsc").
Result(),
backup: builder.ForBackup("velero", "backup").Result(),
expectErr: false,
createVS: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
crClient := velerotest.NewFakeControllerRuntimeClient(t)
logger := logrus.StandardLogger()
p := volumeSnapshotDeleteItemAction{log: logger, crClient: crClient}
if test.vs != nil {
vsMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(test.vs)
require.NoError(t, err)
test.item = &unstructured.Unstructured{Object: vsMap}
}
if test.createVS {
require.NoError(t, crClient.Create(context.TODO(), test.vs))
}
err := p.Execute(
&velero.DeleteItemActionExecuteInput{
Item: test.item,
Backup: test.backup,
},
)
if test.expectErr == false {
require.NoError(t, err)
}
})
}
}
func TestVSAppliesTo(t *testing.T) {
p := volumeSnapshotDeleteItemAction{
log: logrus.StandardLogger(),
}
selector, err := p.AppliesTo()
require.NoError(t, err)
require.Equal(
t,
velero.ResourceSelector{
IncludedResources: []string{"volumesnapshots.snapshot.storage.k8s.io"},
},
selector,
)
}
func TestNewVolumeSnapshotDeleteItemAction(t *testing.T) {
logger := logrus.StandardLogger()
crClient := velerotest.NewFakeControllerRuntimeClient(t)
f := &factorymocks.Factory{}
f.On("KubebuilderClient").Return(nil, fmt.Errorf(""))
plugin := NewVolumeSnapshotDeleteItemAction(f)
_, err := plugin(logger)
require.Error(t, err)
f1 := &factorymocks.Factory{}
f1.On("KubebuilderClient").Return(crClient, nil)
plugin1 := NewVolumeSnapshotDeleteItemAction(f1)
_, err1 := plugin1(logger)
require.NoError(t, err1)
}

93
internal/delete/actions/csi/volumesnapshotcontent_action.go
View File

@ -18,19 +18,23 @@ package csi
import ( import (
"context" "context"
"fmt" "time"
"github.com/google/uuid"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/pkg/errors" "github.com/pkg/errors"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
corev1api "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors" apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/util/wait"
crclient "sigs.k8s.io/controller-runtime/pkg/client" crclient "sigs.k8s.io/controller-runtime/pkg/client"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/client" "github.com/vmware-tanzu/velero/pkg/client"
plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common" plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common"
"github.com/vmware-tanzu/velero/pkg/plugin/velero" "github.com/vmware-tanzu/velero/pkg/plugin/velero"
"github.com/vmware-tanzu/velero/pkg/util/csi" "github.com/vmware-tanzu/velero/pkg/util/boolptr"
kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube" kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube"
) )
@ -77,25 +81,55 @@ func (p *volumeSnapshotContentDeleteItemAction) Execute(
p.log.Infof("Deleting VolumeSnapshotContent %s", snapCont.Name) p.log.Infof("Deleting VolumeSnapshotContent %s", snapCont.Name)
if err := csi.SetVolumeSnapshotContentDeletionPolicy( uuid, err := uuid.NewRandom()
snapCont.Name, if err != nil {
p.crClient, p.log.WithError(err).Errorf("Fail to generate the UUID to create VSC %s", snapCont.Name)
return errors.Wrapf(err, "Fail to generate the UUID to create VSC %s", snapCont.Name)
}
snapCont.Name = "vsc-" + uuid.String()
snapCont.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentDelete
snapCont.Spec.Source = snapshotv1api.VolumeSnapshotContentSource{
SnapshotHandle: snapCont.Status.SnapshotHandle,
}
snapCont.Spec.VolumeSnapshotRef = corev1api.ObjectReference{
APIVersion: snapshotv1api.SchemeGroupVersion.String(),
Kind: "VolumeSnapshot",
Namespace: "ns-" + string(snapCont.UID),
Name: "name-" + string(snapCont.UID),
}
snapCont.ResourceVersion = ""
if err := p.crClient.Create(context.TODO(), &snapCont); err != nil {
return errors.Wrapf(err, "fail to create VolumeSnapshotContent %s", snapCont.Name)
}
// Read resource timeout from backup annotation, if not set, use default value.
timeout, err := time.ParseDuration(
input.Backup.Annotations[velerov1api.ResourceTimeoutAnnotation])
if err != nil {
p.log.Warnf("fail to parse resource timeout annotation %s: %s",
input.Backup.Annotations[velerov1api.ResourceTimeoutAnnotation], err.Error())
timeout = 10 * time.Minute
}
p.log.Debugf("resource timeout is set to %s", timeout.String())
interval := 5 * time.Second
// Wait until VSC created and ReadyToUse is true.
if err := wait.PollUntilContextTimeout(
context.Background(),
interval,
timeout,
true,
func(ctx context.Context) (bool, error) {
return checkVSCReadiness(ctx, &snapCont, p.crClient)
},
); err != nil { ); err != nil {
// #4764: Leave a warning when VolumeSnapshotContent cannot be found for deletion. return errors.Wrapf(err, "fail to wait VolumeSnapshotContent %s becomes ready.", snapCont.Name)
// Manual deleting VolumeSnapshotContent can cause this.
// It's tricky for Velero to handle this inconsistency.
// Even if Velero restores the VolumeSnapshotContent, CSI snapshot controller
// may not delete it correctly due to the snapshot represented by VolumeSnapshotContent
// already deleted on cloud provider.
if apierrors.IsNotFound(err) {
p.log.Warnf(
"VolumeSnapshotContent %s of backup %s cannot be found. May leave orphan snapshot %s on cloud provider.",
snapCont.Name, input.Backup.Name, *snapCont.Status.SnapshotHandle)
return nil
}
return errors.Wrapf(err, fmt.Sprintf(
"failed to set DeletionPolicy on volumesnapshotcontent %s. Skipping deletion",
snapCont.Name))
} }
if err := p.crClient.Delete( if err := p.crClient.Delete(
@ -109,6 +143,25 @@ func (p *volumeSnapshotContentDeleteItemAction) Execute(
return nil return nil
} }
var checkVSCReadiness = func(
ctx context.Context,
vsc *snapshotv1api.VolumeSnapshotContent,
client crclient.Client,
) (bool, error) {
tmpVSC := new(snapshotv1api.VolumeSnapshotContent)
if err := client.Get(ctx, crclient.ObjectKeyFromObject(vsc), tmpVSC); err != nil {
return false, errors.Wrapf(
err, "failed to get VolumeSnapshotContent %s", vsc.Name,
)
}
if tmpVSC.Status != nil && boolptr.IsSetToTrue(tmpVSC.Status.ReadyToUse) {
return true, nil
}
return false, nil
}
func NewVolumeSnapshotContentDeleteItemAction( func NewVolumeSnapshotContentDeleteItemAction(
f client.Factory, f client.Factory,
) plugincommon.HandlerInitializer { ) plugincommon.HandlerInitializer {

96
internal/delete/actions/csi/volumesnapshotcontent_action_test.go
View File

@ -22,10 +22,13 @@ import (
"testing" "testing"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/pkg/errors"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime"
crclient "sigs.k8s.io/controller-runtime/pkg/client"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/builder" "github.com/vmware-tanzu/velero/pkg/builder"
@ -37,11 +40,15 @@ import (
func TestVSCExecute(t *testing.T) { func TestVSCExecute(t *testing.T) {
snapshotHandleStr := "test" snapshotHandleStr := "test"
tests := []struct { tests := []struct {
name string name string
item runtime.Unstructured item runtime.Unstructured
vsc *snapshotv1api.VolumeSnapshotContent vsc *snapshotv1api.VolumeSnapshotContent
backup *velerov1api.Backup backup *velerov1api.Backup
createVSC bool function func(
ctx context.Context,
vsc *snapshotv1api.VolumeSnapshotContent,
client crclient.Client,
) (bool, error)
expectErr bool expectErr bool
}{ }{
{ {
@ -62,17 +69,30 @@ func TestVSCExecute(t *testing.T) {
expectErr: false, expectErr: false,
}, },
{ {
name: "VolumeSnapshotContent doesn't exist in the cluster, no error", name: "Normal case, VolumeSnapshot should be deleted",
vsc: builder.ForVolumeSnapshotContent("bar").ObjectMeta(builder.WithLabelsMap(map[string]string{velerov1api.BackupNameLabel: "backup"})).Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleStr}).Result(), vsc: builder.ForVolumeSnapshotContent("bar").ObjectMeta(builder.WithLabelsMap(map[string]string{velerov1api.BackupNameLabel: "backup"})).Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleStr}).Result(),
backup: builder.ForBackup("velero", "backup").Result(), backup: builder.ForBackup("velero", "backup").ObjectMeta(builder.WithAnnotationsMap(map[string]string{velerov1api.ResourceTimeoutAnnotation: "5s"})).Result(),
expectErr: false, expectErr: false,
function: func(
ctx context.Context,
vsc *snapshotv1api.VolumeSnapshotContent,
client crclient.Client,
) (bool, error) {
return true, nil
},
}, },
{ {
name: "Normal case, VolumeSnapshot should be deleted", name: "Normal case, VolumeSnapshot should be deleted",
vsc: builder.ForVolumeSnapshotContent("bar").ObjectMeta(builder.WithLabelsMap(map[string]string{velerov1api.BackupNameLabel: "backup"})).Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleStr}).Result(), vsc: builder.ForVolumeSnapshotContent("bar").ObjectMeta(builder.WithLabelsMap(map[string]string{velerov1api.BackupNameLabel: "backup"})).Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleStr}).Result(),
backup: builder.ForBackup("velero", "backup").Result(), backup: builder.ForBackup("velero", "backup").ObjectMeta(builder.WithAnnotationsMap(map[string]string{velerov1api.ResourceTimeoutAnnotation: "5s"})).Result(),
expectErr: false, expectErr: true,
createVSC: true, function: func(
ctx context.Context,
vsc *snapshotv1api.VolumeSnapshotContent,
client crclient.Client,
) (bool, error) {
return false, errors.Errorf("test error case")
},
}, },
} }
@ -80,6 +100,7 @@ func TestVSCExecute(t *testing.T) {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
crClient := velerotest.NewFakeControllerRuntimeClient(t) crClient := velerotest.NewFakeControllerRuntimeClient(t)
logger := logrus.StandardLogger() logger := logrus.StandardLogger()
checkVSCReadiness = test.function
p := volumeSnapshotContentDeleteItemAction{log: logger, crClient: crClient} p := volumeSnapshotContentDeleteItemAction{log: logger, crClient: crClient}
@ -89,10 +110,6 @@ func TestVSCExecute(t *testing.T) {
test.item = &unstructured.Unstructured{Object: vscMap} test.item = &unstructured.Unstructured{Object: vscMap}
} }
if test.createVSC {
require.NoError(t, crClient.Create(context.TODO(), test.vsc))
}
err := p.Execute( err := p.Execute(
&velero.DeleteItemActionExecuteInput{ &velero.DeleteItemActionExecuteInput{
Item: test.item, Item: test.item,
@ -140,3 +157,54 @@ func TestNewVolumeSnapshotContentDeleteItemAction(t *testing.T) {
_, err1 := plugin1(logger) _, err1 := plugin1(logger)
require.NoError(t, err1) require.NoError(t, err1)
} }
func TestCheckVSCReadiness(t *testing.T) {
tests := []struct {
name string
vsc *snapshotv1api.VolumeSnapshotContent
createVSC bool
expectErr bool
ready bool
}{
{
name: "VSC not exist",
vsc: &snapshotv1api.VolumeSnapshotContent{
ObjectMeta: metav1.ObjectMeta{
Name: "vsc-1",
Namespace: "velero",
},
},
createVSC: false,
expectErr: true,
ready: false,
},
{
name: "VSC not ready",
vsc: &snapshotv1api.VolumeSnapshotContent{
ObjectMeta: metav1.ObjectMeta{
Name: "vsc-1",
Namespace: "velero",
},
},
createVSC: true,
expectErr: false,
ready: false,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ctx := context.TODO()
crClient := velerotest.NewFakeControllerRuntimeClient(t)
if test.createVSC {
require.NoError(t, crClient.Create(ctx, test.vsc))
}
ready, err := checkVSCReadiness(ctx, test.vsc, crClient)
require.Equal(t, test.ready, ready)
if test.expectErr {
require.Error(t, err)
}
})
}
}

1
internal/volume/volumes_information.go
View File

@ -838,6 +838,7 @@ func (t *RestoreVolumeInfoTracker) Result() []*RestoreVolumeInfo {
if csiSnapshot.Spec.Source.VolumeSnapshotContentName != nil { if csiSnapshot.Spec.Source.VolumeSnapshotContentName != nil {
vscName = *csiSnapshot.Spec.Source.VolumeSnapshotContentName vscName = *csiSnapshot.Spec.Source.VolumeSnapshotContentName
} }
volumeInfo := &RestoreVolumeInfo{ volumeInfo := &RestoreVolumeInfo{
PVCNamespace: pvcNS, PVCNamespace: pvcNS,
PVCName: pvcName, PVCName: pvcName,

4
pkg/backup/actions/csi/pvc_action_test.go
View File

@ -124,8 +124,8 @@ func TestExecute(t *testing.T) {
operationID: ".", operationID: ".",
expectedErr: nil, expectedErr: nil,
expectedPVC: builder.ForPersistentVolumeClaim("velero", "testPVC"). expectedPVC: builder.ForPersistentVolumeClaim("velero", "testPVC").
ObjectMeta(builder.WithAnnotations(velerov1api.MustIncludeAdditionalItemAnnotation, "true", velerov1api.DataUploadNameAnnotation, "velero/", velerov1api.VolumeSnapshotLabel, ""), ObjectMeta(builder.WithAnnotations(velerov1api.MustIncludeAdditionalItemAnnotation, "true", velerov1api.DataUploadNameAnnotation, "velero/"),
builder.WithLabels(velerov1api.BackupNameLabel, "test", velerov1api.VolumeSnapshotLabel, "")). builder.WithLabels(velerov1api.BackupNameLabel, "test")).
VolumeName("testPV").StorageClass("testSC").Phase(corev1.ClaimBound).Result(), VolumeName("testPV").StorageClass("testSC").Phase(corev1.ClaimBound).Result(),
}, },
{ {

5
pkg/backup/actions/csi/volumesnapshot_action.go
View File

@ -131,13 +131,13 @@ func (p *volumeSnapshotBackupItemAction) Execute(
backup.Status.Phase == velerov1api.BackupPhaseFinalizingPartiallyFailed { backup.Status.Phase == velerov1api.BackupPhaseFinalizingPartiallyFailed {
p.log. p.log.
WithField("Backup", fmt.Sprintf("%s/%s", backup.Namespace, backup.Name)). WithField("Backup", fmt.Sprintf("%s/%s", backup.Namespace, backup.Name)).
WithField("BackupPhase", backup.Status.Phase).Debugf("Clean VolumeSnapshots.") WithField("BackupPhase", backup.Status.Phase).Debugf("Cleaning VolumeSnapshots.")
if vsc == nil { if vsc == nil {
vsc = &snapshotv1api.VolumeSnapshotContent{} vsc = &snapshotv1api.VolumeSnapshotContent{}
} }
csi.DeleteVolumeSnapshot(*vs, *vsc, backup, p.crClient, p.log) csi.DeleteReadyVolumeSnapshot(*vs, *vsc, p.crClient, p.log)
return item, nil, "", nil, nil return item, nil, "", nil, nil
} }
@ -164,6 +164,7 @@ func (p *volumeSnapshotBackupItemAction) Execute(
annotations[velerov1api.VolumeSnapshotHandleAnnotation] = *vsc.Status.SnapshotHandle annotations[velerov1api.VolumeSnapshotHandleAnnotation] = *vsc.Status.SnapshotHandle
annotations[velerov1api.DriverNameAnnotation] = vsc.Spec.Driver annotations[velerov1api.DriverNameAnnotation] = vsc.Spec.Driver
} }
if vsc.Status.RestoreSize != nil { if vsc.Status.RestoreSize != nil {
annotations[velerov1api.VolumeSnapshotRestoreSize] = resource.NewQuantity( annotations[velerov1api.VolumeSnapshotRestoreSize] = resource.NewQuantity(
*vsc.Status.RestoreSize, resource.BinarySI).String() *vsc.Status.RestoreSize, resource.BinarySI).String()

4
pkg/builder/volume_snapshot_content_builder.go
View File

@ -20,6 +20,7 @@ import (
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
v1 "k8s.io/api/core/v1" v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
) )
// VolumeSnapshotContentBuilder builds VolumeSnapshotContent object. // VolumeSnapshotContentBuilder builds VolumeSnapshotContent object.
@ -60,12 +61,13 @@ func (v *VolumeSnapshotContentBuilder) DeletionPolicy(policy snapshotv1api.Delet
} }
// VolumeSnapshotRef sets the built VolumeSnapshotContent's spec.VolumeSnapshotRef value. // VolumeSnapshotRef sets the built VolumeSnapshotContent's spec.VolumeSnapshotRef value.
func (v *VolumeSnapshotContentBuilder) VolumeSnapshotRef(namespace, name string) *VolumeSnapshotContentBuilder { func (v *VolumeSnapshotContentBuilder) VolumeSnapshotRef(namespace, name, uid string) *VolumeSnapshotContentBuilder {
v.object.Spec.VolumeSnapshotRef = v1.ObjectReference{ v.object.Spec.VolumeSnapshotRef = v1.ObjectReference{
APIVersion: "snapshot.storage.k8s.io/v1", APIVersion: "snapshot.storage.k8s.io/v1",
Kind: "VolumeSnapshot", Kind: "VolumeSnapshot",
Namespace: namespace, Namespace: namespace,
Name: name, Name: name,
UID: types.UID(uid),
} }
return v return v
} }

16
pkg/cmd/server/plugin/plugin.go
View File

@ -138,10 +138,6 @@ func NewCommand(f client.Factory) *cobra.Command {
"velero.io/dataupload-delete", "velero.io/dataupload-delete",
newDateUploadDeleteItemAction(f), newDateUploadDeleteItemAction(f),
). ).
RegisterDeleteItemAction(
"velero.io/csi-volumesnapshot-delete",
newVolumeSnapshotDeleteItemAction(f),
).
RegisterDeleteItemAction( RegisterDeleteItemAction(
"velero.io/csi-volumesnapshotcontent-delete", "velero.io/csi-volumesnapshotcontent-delete",
newVolumeSnapshotContentDeleteItemAction(f), newVolumeSnapshotContentDeleteItemAction(f),
@ -167,12 +163,12 @@ func NewCommand(f client.Factory) *cobra.Command {
newPvcRestoreItemAction(f), newPvcRestoreItemAction(f),
). ).
RegisterRestoreItemActionV2( RegisterRestoreItemActionV2(
"velero.io/csi-volumesnapshot-restorer", constant.PluginCsiVolumeSnapshotRestoreRIA,
newVolumeSnapshotRestoreItemAction(f), newVolumeSnapshotRestoreItemAction(f),
). ).
RegisterRestoreItemActionV2( RegisterRestoreItemActionV2(
"velero.io/csi-volumesnapshotcontent-restorer", "velero.io/csi-volumesnapshotcontent-restorer",
newVolumeSnapshotContentRestoreItemAction, newVolumeSnapshotContentRestoreItemAction(f),
). ).
RegisterRestoreItemActionV2( RegisterRestoreItemActionV2(
"velero.io/csi-volumesnapshotclass-restorer", "velero.io/csi-volumesnapshotclass-restorer",
@ -432,10 +428,6 @@ func newVolumeSnapshotClassBackupItemAction(logger logrus.FieldLogger) (any, err
// DeleteItemAction plugins // DeleteItemAction plugins
func newVolumeSnapshotDeleteItemAction(f client.Factory) plugincommon.HandlerInitializer {
return dia.NewVolumeSnapshotDeleteItemAction(f)
}
func newVolumeSnapshotContentDeleteItemAction(f client.Factory) plugincommon.HandlerInitializer { func newVolumeSnapshotContentDeleteItemAction(f client.Factory) plugincommon.HandlerInitializer {
return dia.NewVolumeSnapshotContentDeleteItemAction(f) return dia.NewVolumeSnapshotContentDeleteItemAction(f)
} }
@ -450,8 +442,8 @@ func newVolumeSnapshotRestoreItemAction(f client.Factory) plugincommon.HandlerIn
return csiria.NewVolumeSnapshotRestoreItemAction(f) return csiria.NewVolumeSnapshotRestoreItemAction(f)
} }
func newVolumeSnapshotContentRestoreItemAction(logger logrus.FieldLogger) (any, error) { func newVolumeSnapshotContentRestoreItemAction(f client.Factory) plugincommon.HandlerInitializer {
return csiria.NewVolumeSnapshotContentRestoreItemAction(logger) return csiria.NewVolumeSnapshotContentRestoreItemAction(f)
} }
func newVolumeSnapshotClassRestoreItemAction(logger logrus.FieldLogger) (any, error) { func newVolumeSnapshotClassRestoreItemAction(logger logrus.FieldLogger) (any, error) {

3
pkg/constant/constant.go
View File

@ -20,5 +20,6 @@ const (
ControllerServerStatusRequest = "server-status-request" ControllerServerStatusRequest = "server-status-request"
ControllerRestoreFinalizer = "restore-finalizer" ControllerRestoreFinalizer = "restore-finalizer"
PluginCSIPVCRestoreRIA = "velero.io/csi-pvc-restorer" PluginCSIPVCRestoreRIA = "velero.io/csi-pvc-restorer"
PluginCsiVolumeSnapshotRestoreRIA = "velero.io/csi-volumesnapshot-restorer"
) )

39
pkg/controller/backup_sync_controller.go
View File

@ -42,7 +42,6 @@ import (
"github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt" "github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt"
"github.com/vmware-tanzu/velero/pkg/util/kube" "github.com/vmware-tanzu/velero/pkg/util/kube"
corev1api "k8s.io/api/core/v1"
ctrl "sigs.k8s.io/controller-runtime" ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client"
) )
@ -263,44 +262,6 @@ func (b *backupSyncReconciler) Reconcile(ctx context.Context, req ctrl.Request)
log.Infof("Created CSI VolumeSnapshotClass %s", vsClass.Name) log.Infof("Created CSI VolumeSnapshotClass %s", vsClass.Name)
} }
} }
log.Info("Syncing CSI volumesnapshotcontents in backup")
snapConts, err := backupStore.GetCSIVolumeSnapshotContents(backupName)
if err != nil {
log.WithError(errors.WithStack(err)).Error("Error getting CSI volumesnapshotcontents for this backup from backup store")
continue
}
log.Infof("Syncing %d CSI volumesnapshotcontents in backup", len(snapConts))
for _, snapCont := range snapConts {
// TODO: Reset ResourceVersion prior to persisting VolumeSnapshotContents
snapCont.ResourceVersion = ""
// Make the VolumeSnapshotContent static
snapCont.Spec.Source = snapshotv1api.VolumeSnapshotContentSource{
SnapshotHandle: snapCont.Status.SnapshotHandle,
}
// Set VolumeSnapshotRef to none exist one, because VolumeSnapshotContent
// validation webhook will check whether name and namespace are nil.
// external-snapshotter needs Source pointing to snapshot and VolumeSnapshot
// reference's UID to nil to determine the VolumeSnapshotContent is deletable.
snapCont.Spec.VolumeSnapshotRef = corev1api.ObjectReference{
APIVersion: snapshotv1api.SchemeGroupVersion.String(),
Kind: "VolumeSnapshot",
Namespace: "ns-" + string(snapCont.UID),
Name: "name-" + string(snapCont.UID),
}
err := b.client.Create(ctx, snapCont, &client.CreateOptions{})
switch {
case err != nil && apierrors.IsAlreadyExists(err):
log.Debugf("volumesnapshotcontent %s already exists in cluster", snapCont.Name)
continue
case err != nil && !apierrors.IsAlreadyExists(err):
log.WithError(errors.WithStack(err)).Errorf("Error syncing volumesnapshotcontent %s into cluster", snapCont.Name)
continue
default:
log.Infof("Created CSI volumesnapshotcontent %s", snapCont.Name)
}
}
} }
} }

20
pkg/controller/restore_finalizer_controller.go
View File

@ -22,31 +22,27 @@ import (
"sync" "sync"
"time" "time"
"k8s.io/apimachinery/pkg/runtime/schema"
"github.com/vmware-tanzu/velero/pkg/constant"
"github.com/vmware-tanzu/velero/pkg/itemoperation"
"github.com/vmware-tanzu/velero/pkg/plugin/velero"
storagev1api "k8s.io/api/storage/v1"
"github.com/pkg/errors" "github.com/pkg/errors"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
v1 "k8s.io/api/core/v1"
storagev1api "k8s.io/api/storage/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors" apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/util/wait"
"k8s.io/utils/clock"
ctrl "sigs.k8s.io/controller-runtime" ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/utils/clock"
"github.com/vmware-tanzu/velero/internal/hook" "github.com/vmware-tanzu/velero/internal/hook"
"github.com/vmware-tanzu/velero/internal/volume" "github.com/vmware-tanzu/velero/internal/volume"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/constant"
"github.com/vmware-tanzu/velero/pkg/itemoperation"
"github.com/vmware-tanzu/velero/pkg/metrics" "github.com/vmware-tanzu/velero/pkg/metrics"
"github.com/vmware-tanzu/velero/pkg/persistence" "github.com/vmware-tanzu/velero/pkg/persistence"
"github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt" "github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt"
"github.com/vmware-tanzu/velero/pkg/plugin/velero"
kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube" kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube"
"github.com/vmware-tanzu/velero/pkg/util/results" "github.com/vmware-tanzu/velero/pkg/util/results"
) )

8
pkg/controller/restore_finalizer_controller_test.go
View File

@ -23,17 +23,13 @@ import (
"testing" "testing"
"time" "time"
"k8s.io/apimachinery/pkg/runtime/schema"
"github.com/vmware-tanzu/velero/pkg/itemoperation"
"github.com/vmware-tanzu/velero/pkg/plugin/velero"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
corev1api "k8s.io/api/core/v1" corev1api "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/types"
testclocks "k8s.io/utils/clock/testing" testclocks "k8s.io/utils/clock/testing"
ctrl "sigs.k8s.io/controller-runtime" ctrl "sigs.k8s.io/controller-runtime"
@ -43,10 +39,12 @@ import (
"github.com/vmware-tanzu/velero/internal/volume" "github.com/vmware-tanzu/velero/internal/volume"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/builder" "github.com/vmware-tanzu/velero/pkg/builder"
"github.com/vmware-tanzu/velero/pkg/itemoperation"
"github.com/vmware-tanzu/velero/pkg/metrics" "github.com/vmware-tanzu/velero/pkg/metrics"
persistencemocks "github.com/vmware-tanzu/velero/pkg/persistence/mocks" persistencemocks "github.com/vmware-tanzu/velero/pkg/persistence/mocks"
"github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt" "github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt"
pluginmocks "github.com/vmware-tanzu/velero/pkg/plugin/mocks" pluginmocks "github.com/vmware-tanzu/velero/pkg/plugin/mocks"
"github.com/vmware-tanzu/velero/pkg/plugin/velero"
velerotest "github.com/vmware-tanzu/velero/pkg/test" velerotest "github.com/vmware-tanzu/velero/pkg/test"
pkgUtilKubeMocks "github.com/vmware-tanzu/velero/pkg/util/kube/mocks" pkgUtilKubeMocks "github.com/vmware-tanzu/velero/pkg/util/kube/mocks"
"github.com/vmware-tanzu/velero/pkg/util/results" "github.com/vmware-tanzu/velero/pkg/util/results"

19
pkg/restore/actions/csi/pvc_action.go
View File

@ -220,8 +220,10 @@ func (p *pvcRestoreItemAction) Execute(
logger.Infof("DataDownload %s/%s is created successfully.", logger.Infof("DataDownload %s/%s is created successfully.",
dataDownload.Namespace, dataDownload.Name) dataDownload.Namespace, dataDownload.Name)
} else { } else {
volumeSnapshotName, ok := pvcFromBackup.Annotations[velerov1api.VolumeSnapshotLabel] targetVSName := ""
if !ok { if vsName, nameOK := pvcFromBackup.Annotations[velerov1api.VolumeSnapshotLabel]; nameOK {
targetVSName = util.GenerateSha256FromRestoreUIDAndVsName(string(input.Restore.UID), vsName)
} else {
logger.Info("Skipping PVCRestoreItemAction for PVC,", logger.Info("Skipping PVCRestoreItemAction for PVC,",
"PVC does not have a CSI VolumeSnapshot.") "PVC does not have a CSI VolumeSnapshot.")
// Make no change in the input PVC. // Make no change in the input PVC.
@ -229,8 +231,9 @@ func (p *pvcRestoreItemAction) Execute(
UpdatedItem: input.Item, UpdatedItem: input.Item,
}, nil }, nil
} }
if err := restoreFromVolumeSnapshot( if err := restoreFromVolumeSnapshot(
&pvc, newNamespace, p.crClient, volumeSnapshotName, logger, &pvc, newNamespace, p.crClient, targetVSName, logger,
); err != nil { ); err != nil {
logger.Errorf("Failed to restore PVC from VolumeSnapshot.") logger.Errorf("Failed to restore PVC from VolumeSnapshot.")
return nil, errors.WithStack(err) return nil, errors.WithStack(err)
@ -502,19 +505,17 @@ func restoreFromVolumeSnapshot(
}, },
vs, vs,
); err != nil { ); err != nil {
return errors.Wrapf(err, return errors.Wrapf(err, "Failed to get Volumesnapshot %s/%s to restore PVC %s/%s",
fmt.Sprintf("Failed to get Volumesnapshot %s/%s to restore PVC %s/%s", newNamespace, volumeSnapshotName, newNamespace, pvc.Name)
newNamespace, volumeSnapshotName, newNamespace, pvc.Name),
)
} }
if _, exists := vs.Annotations[velerov1api.VolumeSnapshotRestoreSize]; exists { if _, exists := vs.Annotations[velerov1api.VolumeSnapshotRestoreSize]; exists {
restoreSize, err := resource.ParseQuantity( restoreSize, err := resource.ParseQuantity(
vs.Annotations[velerov1api.VolumeSnapshotRestoreSize]) vs.Annotations[velerov1api.VolumeSnapshotRestoreSize])
if err != nil { if err != nil {
return errors.Wrapf(err, fmt.Sprintf( return errors.Wrapf(err,
"Failed to parse %s from annotation on Volumesnapshot %s/%s into restore size", "Failed to parse %s from annotation on Volumesnapshot %s/%s into restore size",
vs.Annotations[velerov1api.VolumeSnapshotRestoreSize], vs.Namespace, vs.Name)) vs.Annotations[velerov1api.VolumeSnapshotRestoreSize], vs.Namespace, vs.Name)
} }
// It is possible that the volume provider allocated a larger // It is possible that the volume provider allocated a larger
// capacity volume than what was requested in the backed up PVC. // capacity volume than what was requested in the backed up PVC.

16
pkg/restore/actions/csi/pvc_action_test.go
View File

@ -45,6 +45,7 @@ import (
"github.com/vmware-tanzu/velero/pkg/label" "github.com/vmware-tanzu/velero/pkg/label"
"github.com/vmware-tanzu/velero/pkg/plugin/velero" "github.com/vmware-tanzu/velero/pkg/plugin/velero"
velerotest "github.com/vmware-tanzu/velero/pkg/test" velerotest "github.com/vmware-tanzu/velero/pkg/test"
"github.com/vmware-tanzu/velero/pkg/util"
"github.com/vmware-tanzu/velero/pkg/util/boolptr" "github.com/vmware-tanzu/velero/pkg/util/boolptr"
) )
@ -546,6 +547,7 @@ func TestCancel(t *testing.T) {
} }
func TestExecute(t *testing.T) { func TestExecute(t *testing.T) {
vsName := util.GenerateSha256FromRestoreUIDAndVsName("restoreUID", "vsName")
tests := []struct { tests := []struct {
name string name string
backup *velerov1api.Backup backup *velerov1api.Backup
@ -573,20 +575,22 @@ func TestExecute(t *testing.T) {
{ {
name: "VolumeSnapshot cannot be found", name: "VolumeSnapshot cannot be found",
backup: builder.ForBackup("velero", "testBackup").Result(), backup: builder.ForBackup("velero", "testBackup").Result(),
restore: builder.ForRestore("velero", "testRestore").Backup("testBackup").Result(), restore: builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("restoreUID")).Backup("testBackup").Result(),
pvc: builder.ForPersistentVolumeClaim("velero", "testPVC").ObjectMeta(builder.WithAnnotations(velerov1api.VolumeSnapshotLabel, "testVS")).Result(), pvc: builder.ForPersistentVolumeClaim("velero", "testPVC").ObjectMeta(builder.WithAnnotations(velerov1api.VolumeSnapshotLabel, "vsName")).Result(),
expectedErr: "Failed to get Volumesnapshot velero/testVS to restore PVC velero/testPVC: volumesnapshots.snapshot.storage.k8s.io \"testVS\" not found", expectedErr: fmt.Sprintf("Failed to get Volumesnapshot velero/%s to restore PVC velero/testPVC: volumesnapshots.snapshot.storage.k8s.io \"%s\" not found", vsName, vsName),
}, },
{ {
name: "Restore from VolumeSnapshot", name: "Restore from VolumeSnapshot",
backup: builder.ForBackup("velero", "testBackup").Result(), backup: builder.ForBackup("velero", "testBackup").Result(),
restore: builder.ForRestore("velero", "testRestore").Backup("testBackup").Result(), restore: builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("restoreUID")).Backup("testBackup").Result(),
pvc: builder.ForPersistentVolumeClaim("velero", "testPVC").ObjectMeta(builder.WithAnnotations(velerov1api.VolumeSnapshotLabel, "testVS")). pvc: builder.ForPersistentVolumeClaim("velero", "testPVC").ObjectMeta(builder.WithAnnotations(velerov1api.VolumeSnapshotLabel, "vsName")).
RequestResource(map[corev1api.ResourceName]resource.Quantity{corev1api.ResourceStorage: resource.MustParse("10Gi")}). RequestResource(map[corev1api.ResourceName]resource.Quantity{corev1api.ResourceStorage: resource.MustParse("10Gi")}).
DataSource(&corev1api.TypedLocalObjectReference{APIGroup: &snapshotv1api.SchemeGroupVersion.Group, Kind: "VolumeSnapshot", Name: "testVS"}). DataSource(&corev1api.TypedLocalObjectReference{APIGroup: &snapshotv1api.SchemeGroupVersion.Group, Kind: "VolumeSnapshot", Name: "testVS"}).
DataSourceRef(&corev1api.TypedObjectReference{APIGroup: &snapshotv1api.SchemeGroupVersion.Group, Kind: "VolumeSnapshot", Name: "testVS"}). DataSourceRef(&corev1api.TypedObjectReference{APIGroup: &snapshotv1api.SchemeGroupVersion.Group, Kind: "VolumeSnapshot", Name: "testVS"}).
Result(), Result(),
vs: builder.ForVolumeSnapshot("velero", "testVS").ObjectMeta(builder.WithAnnotations(velerov1api.VolumeSnapshotRestoreSize, "10Gi")).Result(), vs: builder.ForVolumeSnapshot("velero", vsName).ObjectMeta(
builder.WithAnnotations(velerov1api.VolumeSnapshotRestoreSize, "10Gi"),
).Result(),
expectedPVC: builder.ForPersistentVolumeClaim("velero", "testPVC").Result(), expectedPVC: builder.ForPersistentVolumeClaim("velero", "testPVC").Result(),
}, },
{ {

105
pkg/restore/actions/csi/volumesnapshot_action.go
View File

@ -17,24 +17,19 @@ limitations under the License.
package csi package csi
import ( import (
"context"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/pkg/errors" "github.com/pkg/errors"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
core_v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime"
crclient "sigs.k8s.io/controller-runtime/pkg/client" crclient "sigs.k8s.io/controller-runtime/pkg/client"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/client" "github.com/vmware-tanzu/velero/pkg/client"
"github.com/vmware-tanzu/velero/pkg/label"
plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common" plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common"
"github.com/vmware-tanzu/velero/pkg/plugin/velero" "github.com/vmware-tanzu/velero/pkg/plugin/velero"
"github.com/vmware-tanzu/velero/pkg/util"
"github.com/vmware-tanzu/velero/pkg/util/boolptr" "github.com/vmware-tanzu/velero/pkg/util/boolptr"
"github.com/vmware-tanzu/velero/pkg/util/csi"
) )
// volumeSnapshotRestoreItemAction is a Velero restore item // volumeSnapshotRestoreItemAction is a Velero restore item
@ -54,8 +49,7 @@ func (p *volumeSnapshotRestoreItemAction) AppliesTo() (
}, nil }, nil
} }
func resetVolumeSnapshotSpecForRestore( func resetVolumeSnapshotSpecForRestore(vs *snapshotv1api.VolumeSnapshot, vscName *string) {
vs *snapshotv1api.VolumeSnapshot, vscName *string) {
// Spec of the backed-up object used the PVC as the source // Spec of the backed-up object used the PVC as the source
// of the volumeSnapshot. Restore operation will however, // of the volumeSnapshot. Restore operation will however,
// restore the VolumeSnapshot from the VolumeSnapshotContent // restore the VolumeSnapshot from the VolumeSnapshotContent
@ -68,10 +62,6 @@ func resetVolumeSnapshotAnnotation(vs *snapshotv1api.VolumeSnapshot) {
string(snapshotv1api.VolumeSnapshotContentRetain) string(snapshotv1api.VolumeSnapshotContentRetain)
} }
// Execute uses the data such as CSI driver name, storage
// snapshot handle, snapshot deletion secret (if any) from
// the annotations to recreate a VolumeSnapshotContent object
// and statically bind the VolumeSnapshot object being restored.
func (p *volumeSnapshotRestoreItemAction) Execute( func (p *volumeSnapshotRestoreItemAction) Execute(
input *velero.RestoreItemActionExecuteInput, input *velero.RestoreItemActionExecuteInput,
) (*velero.RestoreItemActionExecuteOutput, error) { ) (*velero.RestoreItemActionExecuteOutput, error) {
@ -90,84 +80,29 @@ func (p *volumeSnapshotRestoreItemAction) Execute(
errors.Wrapf(err, "failed to convert input.Item from unstructured") errors.Wrapf(err, "failed to convert input.Item from unstructured")
} }
// If cross-namespace restore is configured, change the namespace var vsFromBackup snapshotv1api.VolumeSnapshot
// for VolumeSnapshot object to be restored if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
newNamespace, ok := input.Restore.Spec.NamespaceMapping[vs.GetNamespace()] input.ItemFromBackup.UnstructuredContent(), &vsFromBackup); err != nil {
if !ok { return &velero.RestoreItemActionExecuteOutput{},
// Use original namespace errors.Wrapf(err, "failed to convert input.Item from unstructured")
newNamespace = vs.Namespace
} }
if !csi.IsVolumeSnapshotExists(newNamespace, vs.Name, p.crClient) { generatedName := util.GenerateSha256FromRestoreUIDAndVsName(string(input.Restore.UID), vsFromBackup.Name)
snapHandle, exists := vs.Annotations[velerov1api.VolumeSnapshotHandleAnnotation]
if !exists {
return nil, errors.Errorf(
"VolumeSnapshot %s/%s does not have a %s annotation",
vs.Namespace,
vs.Name,
velerov1api.VolumeSnapshotHandleAnnotation,
)
}
csiDriverName, exists := vs.Annotations[velerov1api.DriverNameAnnotation] // Reset Spec to convert the VolumeSnapshot from using
if !exists { // the dynamic VolumeSnapshotContent to the static one.
return nil, errors.Errorf( resetVolumeSnapshotSpecForRestore(&vs, &generatedName)
"VolumeSnapshot %s/%s does not have a %s annotation", // Also reset the VS name to avoid potential conflict caused by multiple restores of the same backup.
vs.Namespace, vs.Name, velerov1api.DriverNameAnnotation) // Both VS and VSC share the same generated name.
} vs.Name = generatedName
p.log.Debugf("Set VolumeSnapshotContent %s/%s DeletionPolicy to Retain to make sure VS deletion in namespace will not delete Snapshot on cloud provider.", // Reset VolumeSnapshot annotation. By now, only change
newNamespace, vs.Name) // DeletionPolicy to Retain.
resetVolumeSnapshotAnnotation(&vs)
vsc := snapshotv1api.VolumeSnapshotContent{
ObjectMeta: metav1.ObjectMeta{
GenerateName: vs.Name + "-",
Labels: map[string]string{
velerov1api.RestoreNameLabel: label.GetValidName(input.Restore.Name),
},
},
Spec: snapshotv1api.VolumeSnapshotContentSpec{
DeletionPolicy: snapshotv1api.VolumeSnapshotContentRetain,
Driver: csiDriverName,
VolumeSnapshotRef: core_v1.ObjectReference{
APIVersion: "snapshot.storage.k8s.io/v1",
Kind: "VolumeSnapshot",
Namespace: newNamespace,
Name: vs.Name,
},
Source: snapshotv1api.VolumeSnapshotContentSource{
SnapshotHandle: &snapHandle,
},
},
}
// we create the VolumeSnapshotContent here instead of relying on the
// restore flow because we want to statically bind this VolumeSnapshot
// with a VolumeSnapshotContent that will be used as its source for pre-populating
// the volume that will be created as a result of the restore. To perform
// this static binding, a bi-directional link between the VolumeSnapshotContent
// and VolumeSnapshot objects have to be setup. Further, it is disallowed
// to convert a dynamically created VolumeSnapshotContent for static binding.
// See: https://github.com/kubernetes-csi/external-snapshotter/issues/274
if err := p.crClient.Create(context.TODO(), &vsc); err != nil {
return nil, errors.Wrapf(err,
"failed to create VolumeSnapshotContents %s",
vsc.GenerateName)
}
p.log.Infof("Created VolumesnapshotContents %s with static binding to volumesnapshot %s/%s",
vsc, newNamespace, vs.Name)
// Reset Spec to convert the VolumeSnapshot from using
// the dynamic VolumeSnapshotContent to the static one.
resetVolumeSnapshotSpecForRestore(&vs, &vsc.Name)
// Reset VolumeSnapshot annotation. By now, only change
// DeletionPolicy to Retain.
resetVolumeSnapshotAnnotation(&vs)
}
vsMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&vs) vsMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&vs)
if err != nil { if err != nil {
p.log.Errorf("Fail to convert VS %s to unstructured", vs.Namespace+"/"+vs.Name)
return nil, errors.WithStack(err) return nil, errors.WithStack(err)
} }
@ -195,6 +130,7 @@ func (p *volumeSnapshotRestoreItemAction) Cancel(
operationID string, operationID string,
restore *velerov1api.Restore, restore *velerov1api.Restore,
) error { ) error {
// CSI Specification doesn't support canceling a snapshot creation.
return nil return nil
} }
@ -206,7 +142,8 @@ func (p *volumeSnapshotRestoreItemAction) AreAdditionalItemsReady(
} }
func NewVolumeSnapshotRestoreItemAction( func NewVolumeSnapshotRestoreItemAction(
f client.Factory) plugincommon.HandlerInitializer { f client.Factory,
) plugincommon.HandlerInitializer {
return func(logger logrus.FieldLogger) (any, error) { return func(logger logrus.FieldLogger) (any, error) {
crClient, err := f.KubebuilderClient() crClient, err := f.KubebuilderClient()
if err != nil { if err != nil {

92
pkg/restore/actions/csi/volumesnapshot_action_test.go
View File

@ -21,8 +21,6 @@ import (
"fmt" "fmt"
"testing" "testing"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@ -36,6 +34,7 @@ import (
factorymocks "github.com/vmware-tanzu/velero/pkg/client/mocks" factorymocks "github.com/vmware-tanzu/velero/pkg/client/mocks"
"github.com/vmware-tanzu/velero/pkg/plugin/velero" "github.com/vmware-tanzu/velero/pkg/plugin/velero"
velerotest "github.com/vmware-tanzu/velero/pkg/test" velerotest "github.com/vmware-tanzu/velero/pkg/test"
"github.com/vmware-tanzu/velero/pkg/util"
) )
var ( var (
@ -89,7 +88,7 @@ func TestResetVolumeSnapshotSpecForRestore(t *testing.T) {
before := tc.vs.DeepCopy() before := tc.vs.DeepCopy()
resetVolumeSnapshotSpecForRestore(&tc.vs, &tc.vscName) resetVolumeSnapshotSpecForRestore(&tc.vs, &tc.vscName)
assert.Equalf(t, tc.vs.Name, before.Name, "unexpected change to Object.Name, Want: %s; Got %s", tc.name, before.Name, tc.vs.Name) assert.Equalf(t, tc.vs.Name, before.Name, "unexpected change to Object.Name, Want: %s; Got %s", before.Name, tc.vs.Name)
assert.Equal(t, tc.vs.Namespace, before.Namespace, "unexpected change to Object.Namespace, Want: %s; Got %s", tc.name, before.Namespace, tc.vs.Namespace) assert.Equal(t, tc.vs.Namespace, before.Namespace, "unexpected change to Object.Namespace, Want: %s; Got %s", tc.name, before.Namespace, tc.vs.Namespace)
assert.NotNil(t, tc.vs.Spec.Source) assert.NotNil(t, tc.vs.Spec.Source)
assert.Nil(t, tc.vs.Spec.Source.PersistentVolumeClaimName) assert.Nil(t, tc.vs.Spec.Source.PersistentVolumeClaimName)
@ -103,15 +102,15 @@ func TestResetVolumeSnapshotSpecForRestore(t *testing.T) {
} }
func TestVSExecute(t *testing.T) { func TestVSExecute(t *testing.T) {
snapshotHandle := "vsc" newVscName := util.GenerateSha256FromRestoreUIDAndVsName("restoreUID", "vsName")
tests := []struct { tests := []struct {
name string name string
item runtime.Unstructured item runtime.Unstructured
vs *snapshotv1api.VolumeSnapshot vs *snapshotv1api.VolumeSnapshot
restore *velerov1api.Restore restore *velerov1api.Restore
expectErr bool expectErr bool
createVS bool createVS bool
expectedVSC *snapshotv1api.VolumeSnapshotContent expectedVS *snapshotv1api.VolumeSnapshot
}{ }{
{ {
name: "Restore's RestorePVs is false", name: "Restore's RestorePVs is false",
@ -119,45 +118,24 @@ func TestVSExecute(t *testing.T) {
expectErr: false, expectErr: false,
}, },
{ {
name: "Namespace remapping and VS already exists in cluster. Nothing change", name: "VS doesn't have VSC in status",
vs: builder.ForVolumeSnapshot("ns", "name").Result(), vs: builder.ForVolumeSnapshot("ns", "name").ObjectMeta(builder.WithAnnotations("1", "1")).Status().Result(),
restore: builder.ForRestore("velero", "restore").NamespaceMappings("ns", "newNS").Result(),
createVS: true,
expectErr: false,
},
{
name: "VS doesn't have VolumeSnapshotHandleAnnotation annotation",
vs: builder.ForVolumeSnapshot("ns", "name").Result(),
restore: builder.ForRestore("velero", "restore").NamespaceMappings("ns", "newNS").Result(),
expectErr: true,
},
{
name: "VS doesn't have DriverNameAnnotation annotation",
vs: builder.ForVolumeSnapshot("ns", "name").ObjectMeta(
builder.WithAnnotations(velerov1api.VolumeSnapshotHandleAnnotation, ""),
).Result(),
restore: builder.ForRestore("velero", "restore").NamespaceMappings("ns", "newNS").Result(), restore: builder.ForRestore("velero", "restore").NamespaceMappings("ns", "newNS").Result(),
expectErr: true, expectErr: true,
}, },
{ {
name: "Normal case, VSC should be created", name: "Normal case, VSC should be created",
vs: builder.ForVolumeSnapshot("ns", "test").ObjectMeta(builder.WithAnnotationsMap( vs: builder.ForVolumeSnapshot("ns", "vsName").ObjectMeta(
map[string]string{ builder.WithAnnotationsMap(
velerov1api.VolumeSnapshotHandleAnnotation: "vsc", map[string]string{
velerov1api.DriverNameAnnotation: "pd.csi.storage.gke.io", velerov1api.VolumeSnapshotHandleAnnotation: "vsc",
}, velerov1api.DriverNameAnnotation: "pd.csi.storage.gke.io",
)).Result(), },
restore: builder.ForRestore("velero", "restore").Result(), ),
expectErr: false, ).SourceVolumeSnapshotContentName(newVscName).Status().BoundVolumeSnapshotContentName("vscName").Result(),
expectedVSC: builder.ForVolumeSnapshotContent("vsc").ObjectMeta( restore: builder.ForRestore("velero", "restore").ObjectMeta(builder.WithUID("restoreUID")).Result(),
builder.WithLabels(velerov1api.RestoreNameLabel, "restore"), expectErr: false,
).VolumeSnapshotRef("ns", "test"). expectedVS: builder.ForVolumeSnapshot("ns", "test").SourceVolumeSnapshotContentName(newVscName).Result(),
DeletionPolicy(snapshotv1api.VolumeSnapshotContentRetain).
Driver("pd.csi.storage.gke.io").
Source(snapshotv1api.VolumeSnapshotContentSource{
SnapshotHandle: &snapshotHandle,
}).
Result(),
}, },
} }
@ -181,10 +159,11 @@ func TestVSExecute(t *testing.T) {
} }
} }
_, err := p.Execute( result, err := p.Execute(
&velero.RestoreItemActionExecuteInput{ &velero.RestoreItemActionExecuteInput{
Item: test.item, Item: test.item,
Restore: test.restore, ItemFromBackup: test.item,
Restore: test.restore,
}, },
) )
@ -192,18 +171,11 @@ func TestVSExecute(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
} }
if test.expectedVSC != nil { if test.expectedVS != nil {
vscList := new(snapshotv1api.VolumeSnapshotContentList) var vs snapshotv1api.VolumeSnapshot
require.NoError(t, p.crClient.List(context.TODO(), vscList)) require.NoError(t, runtime.DefaultUnstructuredConverter.FromUnstructured(
require.True(t, cmp.Equal( result.UpdatedItem.UnstructuredContent(), &vs))
*test.expectedVSC, require.Equal(t, test.expectedVS.Spec, vs.Spec)
vscList.Items[0],
cmpopts.IgnoreFields(
snapshotv1api.VolumeSnapshotContent{},
"Kind", "APIVersion", "GenerateName", "Name",
"ResourceVersion",
),
))
} }
}) })
} }

78
pkg/restore/actions/csi/volumesnapshotcontent_action.go
View File

@ -20,11 +20,16 @@ import (
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
"github.com/pkg/errors" "github.com/pkg/errors"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/runtime/schema"
crclient "sigs.k8s.io/controller-runtime/pkg/client"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/client"
plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common"
"github.com/vmware-tanzu/velero/pkg/plugin/velero" "github.com/vmware-tanzu/velero/pkg/plugin/velero"
"github.com/vmware-tanzu/velero/pkg/util"
"github.com/vmware-tanzu/velero/pkg/util/boolptr" "github.com/vmware-tanzu/velero/pkg/util/boolptr"
"github.com/vmware-tanzu/velero/pkg/util/csi" "github.com/vmware-tanzu/velero/pkg/util/csi"
) )
@ -32,7 +37,8 @@ import (
// volumeSnapshotContentRestoreItemAction is a restore item action // volumeSnapshotContentRestoreItemAction is a restore item action
// plugin for Velero // plugin for Velero
type volumeSnapshotContentRestoreItemAction struct { type volumeSnapshotContentRestoreItemAction struct {
log logrus.FieldLogger log logrus.FieldLogger
client crclient.Client
} }
// AppliesTo returns information indicating VolumeSnapshotContentRestoreItemAction // AppliesTo returns information indicating VolumeSnapshotContentRestoreItemAction
@ -51,34 +57,77 @@ func (p *volumeSnapshotContentRestoreItemAction) AppliesTo() (
func (p *volumeSnapshotContentRestoreItemAction) Execute( func (p *volumeSnapshotContentRestoreItemAction) Execute(
input *velero.RestoreItemActionExecuteInput, input *velero.RestoreItemActionExecuteInput,
) (*velero.RestoreItemActionExecuteOutput, error) { ) (*velero.RestoreItemActionExecuteOutput, error) {
p.log.Info("Starting VolumeSnapshotContentRestoreItemAction")
var snapCont snapshotv1api.VolumeSnapshotContent
if boolptr.IsSetToFalse(input.Restore.Spec.RestorePVs) { if boolptr.IsSetToFalse(input.Restore.Spec.RestorePVs) {
p.log.Infof("Restore did not request for PVs to be restored %s/%s", p.log.Infof("Restore did not request for PVs to be restored %s/%s",
input.Restore.Namespace, input.Restore.Name) input.Restore.Namespace, input.Restore.Name)
return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil
} }
p.log.Info("Starting VolumeSnapshotContentRestoreItemAction")
var vsc snapshotv1api.VolumeSnapshotContent
if err := runtime.DefaultUnstructuredConverter.FromUnstructured( if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
input.Item.UnstructuredContent(), &snapCont); err != nil { input.Item.UnstructuredContent(), &vsc); err != nil {
return &velero.RestoreItemActionExecuteOutput{}, return &velero.RestoreItemActionExecuteOutput{},
errors.Wrapf(err, "failed to convert input.Item from unstructured") errors.Wrapf(err, "failed to convert input.Item from unstructured")
} }
var vscFromBackup snapshotv1api.VolumeSnapshotContent
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(
input.ItemFromBackup.UnstructuredContent(), &vscFromBackup); err != nil {
return &velero.RestoreItemActionExecuteOutput{},
errors.Errorf(err.Error(), "failed to convert input.ItemFromBackup from unstructured")
}
// If cross-namespace restore is configured, change the namespace
// for VolumeSnapshot object to be restored
newNamespace, ok := input.Restore.Spec.NamespaceMapping[vsc.Spec.VolumeSnapshotRef.Namespace]
if ok {
// Update the referenced VS namespace to the mapping one.
vsc.Spec.VolumeSnapshotRef.Namespace = newNamespace
}
// Reset VSC name to align with VS.
vsc.Name = util.GenerateSha256FromRestoreUIDAndVsName(
string(input.Restore.UID), vscFromBackup.Spec.VolumeSnapshotRef.Name)
// Also reset the referenced VS name.
vsc.Spec.VolumeSnapshotRef.Name = vsc.Name
// Reset the ResourceVersion and UID of referenced VolumeSnapshot.
vsc.Spec.VolumeSnapshotRef.ResourceVersion = ""
vsc.Spec.VolumeSnapshotRef.UID = ""
// Set the DeletionPolicy to Retain to avoid VS deletion will not trigger snapshot deletion
vsc.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentRetain
if vscFromBackup.Status != nil && vscFromBackup.Status.SnapshotHandle != nil {
vsc.Spec.Source.VolumeHandle = nil
vsc.Spec.Source.SnapshotHandle = vscFromBackup.Status.SnapshotHandle
} else {
p.log.Errorf("fail to get snapshot handle from VSC %s status", vsc.Name)
return nil, errors.Errorf("fail to get snapshot handle from VSC %s status", vsc.Name)
}
additionalItems := []velero.ResourceIdentifier{} additionalItems := []velero.ResourceIdentifier{}
if csi.IsVolumeSnapshotContentHasDeleteSecret(&snapCont) { if csi.IsVolumeSnapshotContentHasDeleteSecret(&vsc) {
additionalItems = append(additionalItems, additionalItems = append(additionalItems,
velero.ResourceIdentifier{ velero.ResourceIdentifier{
GroupResource: schema.GroupResource{Group: "", Resource: "secrets"}, GroupResource: schema.GroupResource{Group: "", Resource: "secrets"},
Name: snapCont.Annotations[velerov1api.PrefixedSecretNameAnnotation], Name: vsc.Annotations[velerov1api.PrefixedSecretNameAnnotation],
Namespace: snapCont.Annotations[velerov1api.PrefixedSecretNamespaceAnnotation], Namespace: vsc.Annotations[velerov1api.PrefixedSecretNamespaceAnnotation],
}, },
) )
} }
vscMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&vsc)
if err != nil {
return nil, errors.WithStack(err)
}
p.log.Infof("Returning from VolumeSnapshotContentRestoreItemAction with %d additionalItems", p.log.Infof("Returning from VolumeSnapshotContentRestoreItemAction with %d additionalItems",
len(additionalItems)) len(additionalItems))
return &velero.RestoreItemActionExecuteOutput{ return &velero.RestoreItemActionExecuteOutput{
UpdatedItem: input.Item, UpdatedItem: &unstructured.Unstructured{Object: vscMap},
AdditionalItems: additionalItems, AdditionalItems: additionalItems,
}, nil }, nil
} }
@ -108,6 +157,15 @@ func (p *volumeSnapshotContentRestoreItemAction) AreAdditionalItemsReady(
return true, nil return true, nil
} }
func NewVolumeSnapshotContentRestoreItemAction(logger logrus.FieldLogger) (any, error) { func NewVolumeSnapshotContentRestoreItemAction(
return &volumeSnapshotContentRestoreItemAction{logger}, nil f client.Factory,
) plugincommon.HandlerInitializer {
return func(logger logrus.FieldLogger) (any, error) {
crClient, err := f.KubebuilderClient()
if err != nil {
return nil, err
}
return &volumeSnapshotContentRestoreItemAction{logger, crClient}, nil
}
} }

95
pkg/restore/actions/csi/volumesnapshotcontent_action_test.go
View File

@ -17,6 +17,8 @@ limitations under the License.
package csi package csi
import ( import (
"context"
"fmt"
"testing" "testing"
snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
@ -27,18 +29,25 @@ import (
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/builder" "github.com/vmware-tanzu/velero/pkg/builder"
factorymocks "github.com/vmware-tanzu/velero/pkg/client/mocks"
"github.com/vmware-tanzu/velero/pkg/kuberesource" "github.com/vmware-tanzu/velero/pkg/kuberesource"
"github.com/vmware-tanzu/velero/pkg/plugin/velero" "github.com/vmware-tanzu/velero/pkg/plugin/velero"
velerotest "github.com/vmware-tanzu/velero/pkg/test"
"github.com/vmware-tanzu/velero/pkg/util"
) )
func TestVSCExecute(t *testing.T) { func TestVSCExecute(t *testing.T) {
snapshotHandleName := "testHandle"
newVscName := util.GenerateSha256FromRestoreUIDAndVsName("restoreUID", "vsName")
tests := []struct { tests := []struct {
name string name string
item runtime.Unstructured item runtime.Unstructured
vsc *snapshotv1api.VolumeSnapshotContent vsc *snapshotv1api.VolumeSnapshotContent
restore *velerov1api.Restore restore *velerov1api.Restore
expectErr bool expectErr bool
createVSC bool
expectedItems []velero.ResourceIdentifier expectedItems []velero.ResourceIdentifier
expectedVSC *snapshotv1api.VolumeSnapshotContent
}{ }{
{ {
name: "Restore's RestorePVs is false", name: "Restore's RestorePVs is false",
@ -46,14 +55,16 @@ func TestVSCExecute(t *testing.T) {
expectErr: false, expectErr: false,
}, },
{ {
name: "Normal case, additional items should return", name: "Normal case, additional items should return ",
vsc: builder.ForVolumeSnapshotContent("test").ObjectMeta(builder.WithAnnotationsMap( vsc: builder.ForVolumeSnapshotContent("test").ObjectMeta(builder.WithAnnotationsMap(
map[string]string{ map[string]string{
velerov1api.PrefixedSecretNameAnnotation: "name", velerov1api.PrefixedSecretNameAnnotation: "name",
velerov1api.PrefixedSecretNamespaceAnnotation: "namespace", velerov1api.PrefixedSecretNamespaceAnnotation: "namespace",
}, },
)).Result(), )).VolumeSnapshotRef("velero", "vsName", "vsUID").
restore: builder.ForRestore("velero", "restore").Result(), Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleName}).Result(),
restore: builder.ForRestore("velero", "restore").ObjectMeta(builder.WithUID("restoreUID")).
NamespaceMappings("velero", "restore").Result(),
expectErr: false, expectErr: false,
expectedItems: []velero.ResourceIdentifier{ expectedItems: []velero.ResourceIdentifier{
{ {
@ -62,26 +73,63 @@ func TestVSCExecute(t *testing.T) {
Name: "name", Name: "name",
}, },
}, },
expectedVSC: builder.ForVolumeSnapshotContent(newVscName).ObjectMeta(builder.WithAnnotationsMap(
map[string]string{
velerov1api.PrefixedSecretNameAnnotation: "name",
velerov1api.PrefixedSecretNamespaceAnnotation: "namespace",
},
)).VolumeSnapshotRef("restore", newVscName, "").
Source(snapshotv1api.VolumeSnapshotContentSource{SnapshotHandle: &snapshotHandleName}).
DeletionPolicy(snapshotv1api.VolumeSnapshotContentRetain).
Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleName}).Result(),
},
{
name: "VSC exists in cluster, same as the normal case",
vsc: builder.ForVolumeSnapshotContent("test").ObjectMeta(builder.WithAnnotationsMap(
map[string]string{
velerov1api.PrefixedSecretNameAnnotation: "name",
velerov1api.PrefixedSecretNamespaceAnnotation: "namespace",
},
)).VolumeSnapshotRef("velero", "vsName", "vsUID").
Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleName}).Result(),
restore: builder.ForRestore("velero", "restore").ObjectMeta(builder.WithUID("restoreUID")).
NamespaceMappings("velero", "restore").Result(),
createVSC: true,
expectErr: false,
expectedVSC: builder.ForVolumeSnapshotContent(newVscName).ObjectMeta(builder.WithAnnotationsMap(
map[string]string{
velerov1api.PrefixedSecretNameAnnotation: "name",
velerov1api.PrefixedSecretNamespaceAnnotation: "namespace",
},
)).VolumeSnapshotRef("restore", newVscName, "").
Source(snapshotv1api.VolumeSnapshotContentSource{SnapshotHandle: &snapshotHandleName}).
DeletionPolicy(snapshotv1api.VolumeSnapshotContentRetain).
Status(&snapshotv1api.VolumeSnapshotContentStatus{SnapshotHandle: &snapshotHandleName}).Result(),
}, },
} }
for _, test := range tests { for _, test := range tests {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
p, err := NewVolumeSnapshotContentRestoreItemAction(logrus.StandardLogger()) action := volumeSnapshotContentRestoreItemAction{
require.NoError(t, err) log: logrus.StandardLogger(),
client: velerotest.NewFakeControllerRuntimeClient(t),
action := p.(*volumeSnapshotContentRestoreItemAction) }
if test.vsc != nil { if test.vsc != nil {
vsMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(test.vsc) vsMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(test.vsc)
require.NoError(t, err) require.NoError(t, err)
test.item = &unstructured.Unstructured{Object: vsMap} test.item = &unstructured.Unstructured{Object: vsMap}
if test.createVSC {
require.NoError(t, action.client.Create(context.TODO(), test.vsc))
}
} }
output, err := action.Execute( output, err := action.Execute(
&velero.RestoreItemActionExecuteInput{ &velero.RestoreItemActionExecuteInput{
Item: test.item, Item: test.item,
Restore: test.restore, ItemFromBackup: test.item,
Restore: test.restore,
}, },
) )
@ -89,6 +137,18 @@ func TestVSCExecute(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
} }
if test.expectedVSC != nil {
vsc := new(snapshotv1api.VolumeSnapshotContent)
require.NoError(t,
runtime.DefaultUnstructuredConverter.FromUnstructured(
output.UpdatedItem.UnstructuredContent(),
vsc,
),
)
require.Equal(t, test.expectedVSC, vsc)
}
if len(test.expectedItems) > 0 { if len(test.expectedItems) > 0 {
require.Equal(t, test.expectedItems, output.AdditionalItems) require.Equal(t, test.expectedItems, output.AdditionalItems)
} }
@ -112,3 +172,20 @@ func TestVSCAppliesTo(t *testing.T) {
selector, selector,
) )
} }
func TestNewVolumeSnapshotContentRestoreItemAction(t *testing.T) {
logger := logrus.StandardLogger()
crClient := velerotest.NewFakeControllerRuntimeClient(t)
f := &factorymocks.Factory{}
f.On("KubebuilderClient").Return(nil, fmt.Errorf(""))
plugin := NewVolumeSnapshotContentRestoreItemAction(f)
_, err := plugin(logger)
require.Error(t, err)
f1 := &factorymocks.Factory{}
f1.On("KubebuilderClient").Return(crClient, nil)
plugin1 := NewVolumeSnapshotContentRestoreItemAction(f1)
_, err1 := plugin1(logger)
require.NoError(t, err1)
}

167
pkg/util/csi/volume_snapshot.go
View File

@ -70,8 +70,8 @@ func WaitVolumeSnapshotReady(
if err != nil { if err != nil {
return false, errors.Wrapf( return false, errors.Wrapf(
err, err,
fmt.Sprintf("error to get VolumeSnapshot %s/%s", "error to get VolumeSnapshot %s/%s",
volumeSnapshotNS, volumeSnapshot), volumeSnapshotNS, volumeSnapshot,
) )
} }
@ -175,7 +175,7 @@ func EnsureDeleteVS(ctx context.Context, snapshotClient snapshotter.SnapshotV1In
return true, nil return true, nil
} }
return false, errors.Wrapf(err, fmt.Sprintf("error to get VolumeSnapshot %s", vsName)) return false, errors.Wrapf(err, "error to get VolumeSnapshot %s", vsName)
} }
updated = vs updated = vs
@ -234,7 +234,7 @@ func EnsureDeleteVSC(ctx context.Context, snapshotClient snapshotter.SnapshotV1I
return true, nil return true, nil
} }
return false, errors.Wrapf(err, fmt.Sprintf("error to get VolumeSnapshotContent %s", vscName)) return false, errors.Wrapf(err, "error to get VolumeSnapshotContent %s", vscName)
} }
updated = vsc updated = vsc
@ -478,6 +478,7 @@ func IsVolumeSnapshotExists(
func SetVolumeSnapshotContentDeletionPolicy( func SetVolumeSnapshotContentDeletionPolicy(
vscName string, vscName string,
crClient crclient.Client, crClient crclient.Client,
policy snapshotv1api.DeletionPolicy,
) error { ) error {
vsc := new(snapshotv1api.VolumeSnapshotContent) vsc := new(snapshotv1api.VolumeSnapshotContent)
if err := crClient.Get(context.TODO(), crclient.ObjectKey{Name: vscName}, vsc); err != nil { if err := crClient.Get(context.TODO(), crclient.ObjectKey{Name: vscName}, vsc); err != nil {
@ -485,7 +486,7 @@ func SetVolumeSnapshotContentDeletionPolicy(
} }
originVSC := vsc.DeepCopy() originVSC := vsc.DeepCopy()
vsc.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentDelete vsc.Spec.DeletionPolicy = policy
return crClient.Patch(context.TODO(), vsc, crclient.MergeFrom(originVSC)) return crClient.Patch(context.TODO(), vsc, crclient.MergeFrom(originVSC))
} }
@ -515,6 +516,7 @@ func CleanupVolumeSnapshot(
err := SetVolumeSnapshotContentDeletionPolicy( err := SetVolumeSnapshotContentDeletionPolicy(
*vs.Status.BoundVolumeSnapshotContentName, *vs.Status.BoundVolumeSnapshotContentName,
crClient, crClient,
snapshotv1api.VolumeSnapshotContentDelete,
) )
if err != nil { if err != nil {
log.Debugf("Failed to patch DeletionPolicy of volume snapshot %s/%s", log.Debugf("Failed to patch DeletionPolicy of volume snapshot %s/%s",
@ -530,144 +532,46 @@ func CleanupVolumeSnapshot(
} }
} }
// DeleteVolumeSnapshot handles the VolumeSnapshot instance deletion. It will make sure the VolumeSnapshotContent is func DeleteReadyVolumeSnapshot(
// recreated so that the physical snapshot is retained.
func DeleteVolumeSnapshot(
vs snapshotv1api.VolumeSnapshot, vs snapshotv1api.VolumeSnapshot,
vsc snapshotv1api.VolumeSnapshotContent, vsc snapshotv1api.VolumeSnapshotContent,
backup *velerov1api.Backup,
client crclient.Client, client crclient.Client,
logger logrus.FieldLogger, logger logrus.FieldLogger,
) { ) {
modifyVSCFlag := false logger.Infof("Deleting Volumesnapshot %s/%s", vs.Namespace, vs.Name)
if vs.Status != nil && if vs.Status == nil ||
vs.Status.BoundVolumeSnapshotContentName != nil && vs.Status.BoundVolumeSnapshotContentName == nil ||
len(*vs.Status.BoundVolumeSnapshotContentName) > 0 && len(*vs.Status.BoundVolumeSnapshotContentName) <= 0 {
vsc.Spec.DeletionPolicy == snapshotv1api.VolumeSnapshotContentDelete {
modifyVSCFlag = true
} else {
logger.Errorf("VolumeSnapshot %s/%s is not ready. This is not expected.", logger.Errorf("VolumeSnapshot %s/%s is not ready. This is not expected.",
vs.Namespace, vs.Name) vs.Namespace, vs.Name)
return
} }
// Change VolumeSnapshotContent's DeletionPolicy to Retain before deleting VolumeSnapshot, if vs.Status != nil && vs.Status.BoundVolumeSnapshotContentName != nil {
// because VolumeSnapshotContent will be deleted by deleting VolumeSnapshot, when // Patch the DeletionPolicy of the VolumeSnapshotContent to set it to Retain.
// DeletionPolicy is set to Delete, but Velero needs VSC for cleaning snapshot on cloud // This ensures that the volume snapshot in the storage provider is kept.
// in backup deletion. if err := SetVolumeSnapshotContentDeletionPolicy(
if modifyVSCFlag { vsc.Name,
logger.Debugf("Patching VolumeSnapshotContent %s", vsc.Name) client,
originVSC := vsc.DeepCopy() snapshotv1api.VolumeSnapshotContentRetain,
vsc.Spec.DeletionPolicy = snapshotv1api.VolumeSnapshotContentRetain ); err != nil {
err := client.Patch( logger.Warnf("Failed to patch DeletionPolicy of VolumeSnapshot %s/%s",
context.Background(), vs.Namespace, vs.Name)
&vsc,
crclient.MergeFrom(originVSC),
)
if err != nil {
logger.Errorf(
"fail to modify VolumeSnapshotContent %s DeletionPolicy to Retain: %s",
vsc.Name, err.Error(),
)
return return
} }
defer func() { if err := client.Delete(context.TODO(), &vsc); err != nil {
logger.Debugf("Start to recreate VolumeSnapshotContent %s", vsc.Name) logger.WithError(err).Warnf("Failed to delete the VolumeSnapshotContent %s", vsc.Name)
err := recreateVolumeSnapshotContent(vsc, backup, client, logger) }
if err != nil {
logger.Errorf(
"fail to recreate VolumeSnapshotContent %s: %s",
vsc.Name,
err.Error(),
)
}
}()
} }
if err := client.Delete(context.TODO(), &vs); err != nil {
// Delete VolumeSnapshot from cluster logger.WithError(err).Warnf("Failed to delete VolumeSnapshot %s", vs.Namespace+"/"+vs.Name)
logger.Debugf("Deleting VolumeSnapshot %s/%s", vs.Namespace, vs.Name) } else {
err := client.Delete(context.TODO(), &vs) logger.Infof("Deleted VolumeSnapshot %s and VolumeSnapshotContent %s",
if err != nil { vs.Namespace+"/"+vs.Name, vsc.Name)
logger.Errorf("fail to delete VolumeSnapshot %s/%s: %s",
vs.Namespace, vs.Name, err.Error())
} }
} }
// recreateVolumeSnapshotContent will delete then re-create VolumeSnapshotContent,
// because some parameter in VolumeSnapshotContent Spec is immutable,
// e.g. VolumeSnapshotRef and Source.
// Source is updated to let csi-controller thinks the VSC is statically
// provisioned with VS.
// Set VolumeSnapshotRef's UID to nil will let the csi-controller finds out
// the related VS is gone, then VSC can be deleted.
func recreateVolumeSnapshotContent(
vsc snapshotv1api.VolumeSnapshotContent,
backup *velerov1api.Backup,
client crclient.Client,
log logrus.FieldLogger,
) error {
// Read resource timeout from backup annotation, if not set, use default value.
timeout, err := time.ParseDuration(
backup.Annotations[velerov1api.ResourceTimeoutAnnotation])
if err != nil {
log.Warnf("fail to parse resource timeout annotation %s: %s",
backup.Annotations[velerov1api.ResourceTimeoutAnnotation], err.Error())
timeout = 10 * time.Minute
}
log.Debugf("resource timeout is set to %s", timeout.String())
interval := 1 * time.Second
if err := client.Delete(context.TODO(), &vsc); err != nil {
return errors.Wrapf(err, "fail to delete VolumeSnapshotContent: %s", vsc.Name)
}
// Check VolumeSnapshotContents is already deleted, before re-creating it.
err = wait.PollUntilContextTimeout(
context.Background(),
interval,
timeout,
true,
func(ctx context.Context) (bool, error) {
tmpVSC := new(snapshotv1api.VolumeSnapshotContent)
if err := client.Get(ctx, crclient.ObjectKeyFromObject(&vsc), tmpVSC); err != nil {
if apierrors.IsNotFound(err) {
return true, nil
}
return false, errors.Wrapf(
err,
fmt.Sprintf("failed to get VolumeSnapshotContent %s", vsc.Name),
)
}
return false, nil
},
)
if err != nil {
return errors.Wrapf(err, "fail to retrieve VolumeSnapshotContent %s info", vsc.Name)
}
// Make the VolumeSnapshotContent static
vsc.Spec.Source = snapshotv1api.VolumeSnapshotContentSource{
SnapshotHandle: vsc.Status.SnapshotHandle,
}
// Set VolumeSnapshotRef to none exist one, because VolumeSnapshotContent
// validation webhook will check whether name and namespace are nil.
// external-snapshotter needs Source pointing to snapshot and VolumeSnapshot
// reference's UID to nil to determine the VolumeSnapshotContent is deletable.
vsc.Spec.VolumeSnapshotRef = corev1api.ObjectReference{
APIVersion: snapshotv1api.SchemeGroupVersion.String(),
Kind: "VolumeSnapshot",
Namespace: "ns-" + string(vsc.UID),
Name: "name-" + string(vsc.UID),
}
// ResourceVersion shouldn't exist for new creation.
vsc.ResourceVersion = ""
if err := client.Create(context.TODO(), &vsc); err != nil {
return errors.Wrapf(err, "fail to create VolumeSnapshotContent %s", vsc.Name)
}
return nil
}
// WaitUntilVSCHandleIsReady returns the VolumeSnapshotContent // WaitUntilVSCHandleIsReady returns the VolumeSnapshotContent
// object associated with the volumesnapshot // object associated with the volumesnapshot
func WaitUntilVSCHandleIsReady( func WaitUntilVSCHandleIsReady(
@ -718,9 +622,10 @@ func WaitUntilVSCHandleIsReady(
vs, vs,
); err != nil { ); err != nil {
return false, return false,
errors.Wrapf(err, fmt.Sprintf( errors.Wrapf(
err,
"failed to get volumesnapshot %s/%s", "failed to get volumesnapshot %s/%s",
volSnap.Namespace, volSnap.Name), volSnap.Namespace, volSnap.Name,
) )
} }
@ -740,8 +645,8 @@ func WaitUntilVSCHandleIsReady(
return false, return false,
errors.Wrapf( errors.Wrapf(
err, err,
fmt.Sprintf("failed to get VolumeSnapshotContent %s for VolumeSnapshot %s/%s", "failed to get VolumeSnapshotContent %s for VolumeSnapshot %s/%s",
*vs.Status.BoundVolumeSnapshotContentName, vs.Namespace, vs.Name), *vs.Status.BoundVolumeSnapshotContentName, vs.Namespace, vs.Name,
) )
} }

45
pkg/util/csi/volume_snapshot_test.go
View File

@ -1379,12 +1379,14 @@ func TestSetVolumeSnapshotContentDeletionPolicy(t *testing.T) {
testCases := []struct { testCases := []struct {
name string name string
inputVSCName string inputVSCName string
policy snapshotv1api.DeletionPolicy
objs []runtime.Object objs []runtime.Object
expectError bool expectError bool
}{ }{
{ {
name: "should update DeletionPolicy of a VSC from retain to delete", name: "should update DeletionPolicy of a VSC from retain to delete",
inputVSCName: "retainVSC", inputVSCName: "retainVSC",
policy: snapshotv1api.VolumeSnapshotContentDelete,
objs: []runtime.Object{ objs: []runtime.Object{
&snapshotv1api.VolumeSnapshotContent{ &snapshotv1api.VolumeSnapshotContent{
ObjectMeta: metav1.ObjectMeta{ ObjectMeta: metav1.ObjectMeta{
@ -1400,6 +1402,7 @@ func TestSetVolumeSnapshotContentDeletionPolicy(t *testing.T) {
{ {
name: "should be a no-op updating if DeletionPolicy of a VSC is already Delete", name: "should be a no-op updating if DeletionPolicy of a VSC is already Delete",
inputVSCName: "deleteVSC", inputVSCName: "deleteVSC",
policy: snapshotv1api.VolumeSnapshotContentDelete,
objs: []runtime.Object{ objs: []runtime.Object{
&snapshotv1api.VolumeSnapshotContent{ &snapshotv1api.VolumeSnapshotContent{
ObjectMeta: metav1.ObjectMeta{ ObjectMeta: metav1.ObjectMeta{
@ -1415,6 +1418,7 @@ func TestSetVolumeSnapshotContentDeletionPolicy(t *testing.T) {
{ {
name: "should update DeletionPolicy of a VSC with no DeletionPolicy", name: "should update DeletionPolicy of a VSC with no DeletionPolicy",
inputVSCName: "nothingVSC", inputVSCName: "nothingVSC",
policy: snapshotv1api.VolumeSnapshotContentDelete,
objs: []runtime.Object{ objs: []runtime.Object{
&snapshotv1api.VolumeSnapshotContent{ &snapshotv1api.VolumeSnapshotContent{
ObjectMeta: metav1.ObjectMeta{ ObjectMeta: metav1.ObjectMeta{
@ -1436,7 +1440,7 @@ func TestSetVolumeSnapshotContentDeletionPolicy(t *testing.T) {
for _, tc := range testCases { for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
fakeClient := velerotest.NewFakeControllerRuntimeClient(t, tc.objs...) fakeClient := velerotest.NewFakeControllerRuntimeClient(t, tc.objs...)
err := SetVolumeSnapshotContentDeletionPolicy(tc.inputVSCName, fakeClient) err := SetVolumeSnapshotContentDeletionPolicy(tc.inputVSCName, fakeClient, tc.policy)
if tc.expectError { if tc.expectError {
assert.Error(t, err) assert.Error(t, err)
} else { } else {
@ -1450,7 +1454,7 @@ func TestSetVolumeSnapshotContentDeletionPolicy(t *testing.T) {
assert.NoError(t, err) assert.NoError(t, err)
assert.Equal( assert.Equal(
t, t,
snapshotv1api.VolumeSnapshotContentDelete, tc.policy,
actual.Spec.DeletionPolicy, actual.Spec.DeletionPolicy,
) )
} }
@ -1460,11 +1464,10 @@ func TestSetVolumeSnapshotContentDeletionPolicy(t *testing.T) {
func TestDeleteVolumeSnapshots(t *testing.T) { func TestDeleteVolumeSnapshots(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
vs snapshotv1api.VolumeSnapshot vs snapshotv1api.VolumeSnapshot
vsc snapshotv1api.VolumeSnapshotContent vsc snapshotv1api.VolumeSnapshotContent
expectedVS snapshotv1api.VolumeSnapshot keepVSAndVSC bool
expectedVSC snapshotv1api.VolumeSnapshotContent
}{ }{
{ {
name: "VS is ReadyToUse, and VS has corresponding VSC. VS should be deleted.", name: "VS is ReadyToUse, and VS has corresponding VSC. VS should be deleted.",
@ -1474,10 +1477,6 @@ func TestDeleteVolumeSnapshots(t *testing.T) {
vsc: *builder.ForVolumeSnapshotContent("vsc1"). vsc: *builder.ForVolumeSnapshotContent("vsc1").
DeletionPolicy(snapshotv1api.VolumeSnapshotContentDelete). DeletionPolicy(snapshotv1api.VolumeSnapshotContentDelete).
Status(&snapshotv1api.VolumeSnapshotContentStatus{}).Result(), Status(&snapshotv1api.VolumeSnapshotContentStatus{}).Result(),
expectedVS: snapshotv1api.VolumeSnapshot{},
expectedVSC: *builder.ForVolumeSnapshotContent("vsc1").
DeletionPolicy(snapshotv1api.VolumeSnapshotContentRetain).
VolumeSnapshotRef("ns-", "name-").Result(),
}, },
{ {
name: "VS status is nil. VSC should not be modified.", name: "VS status is nil. VSC should not be modified.",
@ -1486,9 +1485,7 @@ func TestDeleteVolumeSnapshots(t *testing.T) {
vsc: *builder.ForVolumeSnapshotContent("vsc1"). vsc: *builder.ForVolumeSnapshotContent("vsc1").
DeletionPolicy(snapshotv1api.VolumeSnapshotContentDelete). DeletionPolicy(snapshotv1api.VolumeSnapshotContentDelete).
Status(&snapshotv1api.VolumeSnapshotContentStatus{}).Result(), Status(&snapshotv1api.VolumeSnapshotContentStatus{}).Result(),
expectedVS: snapshotv1api.VolumeSnapshot{}, keepVSAndVSC: true,
expectedVSC: *builder.ForVolumeSnapshotContent("vsc1").
DeletionPolicy(snapshotv1api.VolumeSnapshotContentDelete).Result(),
}, },
} }
@ -1499,10 +1496,8 @@ func TestDeleteVolumeSnapshots(t *testing.T) {
[]runtime.Object{&tc.vs, &tc.vsc}..., []runtime.Object{&tc.vs, &tc.vsc}...,
) )
logger := logging.DefaultLogger(logrus.DebugLevel, logging.FormatText) logger := logging.DefaultLogger(logrus.DebugLevel, logging.FormatText)
backup := builder.ForBackup(velerov1api.DefaultNamespace, "backup-1").
DefaultVolumesToFsBackup(false).Result()
DeleteVolumeSnapshot(tc.vs, tc.vsc, backup, client, logger) DeleteReadyVolumeSnapshot(tc.vs, tc.vsc, client, logger)
vsList := new(snapshotv1api.VolumeSnapshotList) vsList := new(snapshotv1api.VolumeSnapshotList)
err := client.List( err := client.List(
@ -1513,12 +1508,6 @@ func TestDeleteVolumeSnapshots(t *testing.T) {
}, },
) )
require.NoError(t, err) require.NoError(t, err)
if tc.expectedVS.Name == "" {
require.Empty(t, vsList.Items)
} else {
require.Equal(t, tc.expectedVS.Status, vsList.Items[0].Status)
require.Equal(t, tc.expectedVS.Spec, vsList.Items[0].Spec)
}
vscList := new(snapshotv1api.VolumeSnapshotContentList) vscList := new(snapshotv1api.VolumeSnapshotContentList)
err = client.List( err = client.List(
@ -1526,8 +1515,14 @@ func TestDeleteVolumeSnapshots(t *testing.T) {
vscList, vscList,
) )
require.NoError(t, err) require.NoError(t, err)
require.Len(t, vscList.Items, 1)
require.Equal(t, tc.expectedVSC.Spec, vscList.Items[0].Spec) if tc.keepVSAndVSC {
require.Equal(t, crclient.ObjectKeyFromObject(&tc.vs), crclient.ObjectKeyFromObject(&vsList.Items[0]))
require.Equal(t, crclient.ObjectKeyFromObject(&tc.vsc), crclient.ObjectKeyFromObject(&vscList.Items[0]))
} else {
require.Empty(t, vsList.Items)
require.Empty(t, vscList.Items)
}
}) })
} }
} }

12
pkg/util/util.go
View File

@ -16,6 +16,11 @@ limitations under the License.
package util package util
import (
"crypto/sha256"
"encoding/hex"
)
func Contains(slice []string, key string) bool { func Contains(slice []string, key string) bool {
for _, i := range slice { for _, i := range slice {
if i == key { if i == key {
@ -24,3 +29,10 @@ func Contains(slice []string, key string) bool {
} }
return false return false
} }
// GenerateSha256FromRestoreUIDAndVsName Use the restore UID and the VS Name to generate
// the new VSC and VS name. By this way, VS and VSC RIA action can get the same VSC name.
func GenerateSha256FromRestoreUIDAndVsName(restoreUID string, vsName string) string {
sha256Bytes := sha256.Sum256([]byte(restoreUID + "/" + vsName))
return hex.EncodeToString(sha256Bytes[:])
}

4
site/content/docs/main/csi.md
View File

@ -109,8 +109,8 @@ Velero will include the generated VolumeSnapshot and VolumeSnapshotContent objec
upload all VolumeSnapshots and VolumeSnapshotContents objects in a JSON file to the object storage system. **Note that upload all VolumeSnapshots and VolumeSnapshotContents objects in a JSON file to the object storage system. **Note that
only Kubernetes objects are uploaded to the object storage, not the data in snapshots.** only Kubernetes objects are uploaded to the object storage, not the data in snapshots.**
When Velero synchronizes backups into a new cluster, VolumeSnapshotContent objects and the VolumeSnapshotClass that is chosen to take From v1.16, when Velero synchronizes backups into a new cluster, the VolumeSnapshotClass that is chosen to take
snapshot will be synced into the cluster as well, so that Velero can manage backup expiration appropriately. snapshot will be synced into the cluster, so that Velero can manage backup expiration appropriately.
The `DeletionPolicy` on the VolumeSnapshotContent will be the same as the `DeletionPolicy` on the VolumeSnapshotClass that was used to create the VolumeSnapshot. Setting a `DeletionPolicy` of `Retain` on the VolumeSnapshotClass will preserve the volume snapshot in the storage system for the lifetime of the Velero backup and will prevent the deletion of the volume snapshot, in the storage system, in the event of a disaster where the namespace with the VolumeSnapshot object may be lost. The `DeletionPolicy` on the VolumeSnapshotContent will be the same as the `DeletionPolicy` on the VolumeSnapshotClass that was used to create the VolumeSnapshot. Setting a `DeletionPolicy` of `Retain` on the VolumeSnapshotClass will preserve the volume snapshot in the storage system for the lifetime of the Velero backup and will prevent the deletion of the volume snapshot, in the storage system, in the event of a disaster where the namespace with the VolumeSnapshot object may be lost.