1. Add --insecure-tls for ResticManager's commands.
2. Add --insecure-tls in PodVolumeBackup and PodVolumeRestore controller.
3. Upgrade integrated Restic version to v0.13.1
4. Change --last flag in Restic command to --latest=1 due to Restic version update.
Signed-off-by: Xun Jiang <jxun@vmware.com>
As we are refactoring controllers with kubebuilder, use the controller-gen rather than code-generator to generate the deep copy methods for objects
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Add bsl related TTL gc errors to labelSelectors
* if backup label map is nil, make map
* clear label if not BSL error
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
This allows a user inspecting the restore logs to see any
errors or warnings generated by the restore so that they
will be seen even without having to use the describe cli.
Signed-off-by: Scott Seago <sseago@redhat.com>
1. Mark the BSL as "Unavailable" when gets any error
2. Add a new field "Message" to the BSL status to record the error message
Fixes#4485Fixes#4405
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
When velero is running on clusters that don't support v1beta1 CRD, the
plugin will not try to backup v1beta1 CRD.
The plugin should be kept for backward compatibility. It will be
removed when velero drop the support for k8s v1.21
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. rename zoneSeparator to gkeZoneSeparator
2. add example of regional PV's node affinity. modify test case description.
Signed-off-by: Xun Jiang <jxun@vmware.com>
Specify the risk of this parameter set to true. Add the issue first reported about this topic which includeds the google document illustrates about it.
Signed-off-by: Xun Jiang <jxun@vmware.com>
Fix#4499
When hook influnce multiple pods, current logic's first pod's container will overwrite the hook's exec.container parameter. That will cause the other pod fail on the hook executing.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Use OrderedResources in schedules
Make ParseOrderedResources public for use in schedules
Add changelog
Signed-off-by: Dominic <dominic@xdnx.org>
* Rename function in comment section
Signed-off-by: Dominic <dominic@xdnx.org>
We introduces the installation option "--default-restic-prune-frequency" to make restic prune frequency configuration in the previous release, but there is a bug that make the option don't take effect. This commit fixes the bug by removing the evaluation part. The restic repository controller will take care the prune frequency for the repository
Fixes#3062
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
Check the existence of the expected service when ignoring the NodePort already allocated error
Fixes 2308
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Migrate backup sync controller from code-generator to kubebuilder
1. use kubebuilder's reconcile logic to replace controller's old logic.
2. use ginkgo and gomega to replace testing.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Fix: modify code according to comments
1. Remove DefaultBackupLocation
2. Remove unneccessary comment line
3. Add syncPeriod default value setting logic
4. Modify ListBackupStorageLocations function's context parameter
5. Add RequeueAfter parameter in Reconcile function return value
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Reconcile function use context passed from parameter
1. Use context passed from parameter, instead of using Reconciler struct's context.
2. Delete Reconciler struct's context member.
3. Modify test case accordingly.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Remove backups and restic repos associated with deleted BSL(s)
Signed-off-by: F. Gold <fgold@vmware.com>
* add changelog
Signed-off-by: F. Gold <fgold@vmware.com>
* Add PR number to changelog
Signed-off-by: F. Gold <fgold@vmware.com>
* Fix typo
Signed-off-by: F. Gold <fgold@vmware.com>
* Only delete backups and restic repos and report success when without errors
Signed-off-by: F. Gold <fgold@vmware.com>
* Adds <backup-name>-itemsnapshots.gz file to backup (when provided). Also
adds DownloadTargetKindBackupItemSnapshots type to allow downloading.
Updated object store unit test
Fixes#3758
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
* Removed redundant checks
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
* Consolidated code for resolving actions and plugins into ActionResolver. Added BackupWithResolvers and
RestoreWithResolvers. Introduces ItemSnapshooterResolver to bring ItemSnapshotter plugins into backup and
restore. ItemSnapshotters are not used yet.
Added action_resolver_test
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
* Addressed review comments
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
This commit adds a restore action item plugin to reset invalid value
of "sideEffects" in resource of mutatingwebhookconfiguration and
validating webhookconfiguration.
To fix the problem the "sideEffects" is illegal for resource migrated
from v1beta1.
fixes#3516
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. remove config/crd/v1beta1
2. remove PROJECT file
3. update controller-gen and kubebuilder version
4. generate client and CRD file
5. add changelog and remove v1beta1 CRD generated code.
6. add kubebuilder test bundle setup command.
7. due to apiextensions.k8s.io/v1beta1 is not supported, only k8s after v1.16 is supported, so remove v1.15 check.
8. add CRD and k8s suppored version update in changelog.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* fix: modify generated from schedule's backup name timestamp to UTC timezone
fix#4279
When backup is created from schedule, and the backup name is not specified, a containing-timestamp generated name will be used. Due to velero client not set timezone to UTC, a local timezone will be used for the generated name.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* fix: modify generated from schedule's backup name timestamp to UTC timezone
fix#4279
When backup is created from schedule, and the backup name is not specified, a containing-timestamp generated name will be used. Due to velero client not set timezone to UTC, a local timezone will be used for the generated name.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* fix: modify generated from schedule's backup name timestamp to UTC timezone
fix#4279
When backup is created from schedule, and the backup name is not specified, a containing-timestamp generated name will be used. Due to velero client not set timezone to UTC, a local timezone will be used for the generated name.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* modify changelog description
Reword the changelog description according to comments.
Signed-off-by: Xun Jiang <jxun@vmware.com>
Co-authored-by: jxun <jxun@jxun-a01.vmware.com>
Co-authored-by: Xun Jiang <jxun@vmware.com>
logrusr is a open source convertor, which can convert logrus logger into logr.
By using logrusr, velero can use exsiting formatted logrus logger, other than introducing zap as a new logger.
Signed-off-by: Xun Jiang <jxun@vmware.com>
Added ItemSnapshotter.proto
Added item_snapshotter Go interface
Added framework components for item_snapshotter
Updated plugins doc with ItemSnapshotter info
Added SnapshotPhase to item_snapshotter.go
ProgressOutputOutput now includes a phase as well as an error string for problems that occured
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
The error should be returned explicitly, because when the default URL is
used S3 will return a 301 and the response can't be handled by restic.
Fixes#4178
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Previously, the BSL credential field would always be set when using the
`create` command, even if no credential details were provided. This
would result in an empty `SecretKeySelector` in the BSL which would
cause operations using this BSL to fail as Velero would attempt to fetch
a `Secret` with an empty name from the K8s API server.
With this change, the `Credential` field is only set if credential
details have been specified. This change also includes some refactoring
to allow the change to be tested.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
fix paging items in to use list options passed by the paging function
The client-go pager sets the Limit options for the list call
to paginate the request[1]. This PR fixes the paging function
to use the options passed by the pager instead of shadowed options
This is required for the pagination to work correctly.
- simplify the pager list implementation by using pager.List()
The List() function already implements a lot of the logic that was
needed for paging here, using it simplifies the code.
1. 3f40906dd8/staging/src/k8s.io/client-go/tools/pager/pager.go (L219)
Signed-off-by: Alay Patel <alay1431@gmail.com>
Bump up restic to v0.12.1 to fix CVE-2020-26160.
Bump up module "github.com/vmware-tanzu/crash-diagnostics" to v0.3.7 to fix CVE-2020-29652.
The "github.com/vmware-tanzu/crash-diagnostics" updates client-go to v0.22.2 which introduces several break changes, this commit updates the related codes as well
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
The errors of restore/backup may be stored in object storage
The well formatted output of describe is also helpful for debugging.
This commit add the command to the crashd script so the output of
"velero backup/restore describe xxx" can be collected
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Add namespace validation in the client
Signed-off-by: F. Gold <fgold@vmware.com>
* Add namespace validation in the backup controller
Signed-off-by: F. Gold <fgold@vmware.com>
* Add changelog for PR 4057
Signed-off-by: F. Gold <fgold@vmware.com>
* Update Copyright notice
Signed-off-by: F. Gold <fgold@vmware.com>
* Update include_excludes_test.go to follow Go standards and be easier to read
Signed-off-by: F. Gold <fgold@vmware.com>
* Add unit tests for namespace validation functions
Signed-off-by: F. Gold <fgold@vmware.com>
* Make changes per review comments
- use one set of namespace validation logic instead of writing two
- remove duplicate namespace validation functions and tests
- add namespace validation tests in includes_excludes_test.go
Signed-off-by: F. Gold <fgold@vmware.com>
* Return all ns validation err msgs as error list
Signed-off-by: F. Gold <fgold@vmware.com>
* Make error message more clear
Signed-off-by: F. Gold <fgold@vmware.com>
Velero was including DownwardAPI volumes when backing up with restic.
When restoring these volumes, it triggered a known issue with restic (as
seen in #3863). Like projected volumes, these volumes should be skipped
as their contents are populated by the Kubernetes API server.
With this change, we are now skipping the restic backup of volumes with
a DownwardAPI source. We are also skipping the restore of any volume
that had a DownwardAPI source as there will exist backups that were
taken prior to this fix being introduced. This will allow these backups
to be restored succesfully.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
This commit removes `IsUnstructuredCRDReady` since
kubernetes/kubernetes#87675 is fixed.
Is uses `Is1CRDReady` to check the readiness of CRD.
After v1.7 we may consider merge the funcx `IsV1Beta1CRDReady` and
`IsV1CRDReady`
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
1. Support to customize the restic restore helper image
2. Use a seperated context when doing the clean up works
3. Wait a while before doing the the restore for aws to avoid #1799
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
This PR added a subcommand `velero debug`, which leverages `crashd` to
collect logs and specs of velero server components and bundle them in a
tarball.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Wait the namespace deletion completed before removing the CRDs when uninstalling the velero
Fixes#3974
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
a timestamp. If two requests were happening very close together for the
same backup, the second would fail randomly.
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
Instead of converting the unstructured item to check for the presence of
the `kube-aggregator.kubernetes.io/automanaged` label, use this label in
the `AppliesTo` to enable the restore logic to select the item. This
means that any item that matches the selector will have restore skipped.
Also add a new test case to the restore action test to check that label
selectors are applied correctly.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
It was discovered during Velero 1.6.3 upgrade testing that Velero was
restoring `APIService` objects for APIs that are no longer being served
by Kubernetes 1.22. If these items were restored, it would break the
behaviour of discovery within the cluster.
This change introduces a new RestoreItemAction plugin that skips the
restore of any `APIService` object which is managed by Kubernetes such
as those for built-in APIs or CRDs. The `APIService`s for these will be
created when the Kubernetes API server starts or when new CRDs are
registered. These objects are identified by looking for the
`kube-aggregator.kubernetes.io/automanaged` label.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Use appropriate CRD API during readiness check
The readiness check for the Velero CRDs was still using the v1beta1 API.
This would cause the readiness check to fail on 1.22 clusters as the
v1beta1 API is no longer available. Previously, this error would be
ignored and the installation would proceed, however with #4002, we are
no longer ignoring errors from this check.
This change modifies the CRD readiness check to check the CRDs using the
same API version that was used when submitting the CRDs to the cluster.
It also introduces a new CRD builder using the V1 API for testing.
This change also fixes a bug that was identified in the polling code
where if the CRDs were not ready on the first polling iteration, they
would be added again to the list of CRDs to check resulting in
duplicates. This would cause the length check to fail on all subsequent
polls and the timeout would always be reached.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Remove duplicate V1 CRD builder and update comment
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
If the Velero CLI can't discover the Kubernetes preferred CRD API
version, use the flag --crds-version to determine the CRDs version.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
This adds a new `buildinfo` variable `ImageRegistry` that can set at
build time like the `Version` variable. This allows us to customise the
Velero binary to use different registries.
If the variable is set, this variable wille be used when creating the
URIs for both the main `velero` and `velero-restic-restore-helper`
images. If it is not set, default to using Dockerhub (`velero/velero`,
`velero/velero-restic-restore-helper`).
There are numerous ways in which the Velero binary can be built so all
of them have been updated to add the new link time flag to set the
variable:
* `make local` (used for local developer builds to build for the local
OS and ARCH)
* `make build` (used by developers and also VMware internal builds to
build a specific OS and ARCH)
* Goreleaser config (used when creating OSS release binaries)
* Dockerfile (used to build the Velero binary used within the image)
All of these workflows are currently triggered from our Makefile where
the variable `REGISTRY` is already available with the default value of
`velero` and used to build the image tag. Where the new `ImageRegistry`
build variable is needed, we pass through this Makefile variable to
those tasks so it can be used accordingly.
The GitHub action and the `./hack/docker-push.sh` script used to push
container images has not been modified. This will continue to use the
default registry specified in the Makefile and will not explicitly pass
it in.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
In #3863, it was discovered that volumes from projected sources were
being backed up by restic when they should have been skipped. Restoring
these volumes triggers a known bug in restic.
In #3866, we started skipping volumes from a projected source, however
there will exist backups that were taken before this fix was introduced.
This change modifies the restore logic to skip the restore of any volume
that came from a projected source, allowing backups taken before #3866
to be restored successfully.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
phases as part of Upload Progress Monitoring, fixes#3755 Add backup phases
needed for Upload Progress Monitoring
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
Previously `WithPlugins` only supported passing image URIs "by tag" --
e.g. `gcr.io/my-repo/my-image:v0.1.2`. With this commit, we add support
for pulling "by digest" -- e.g.
`gcr.io/my-repo/my-image@sha256:a75f9e8c3ced3943515f249597be389f8233e1258d289b11184796edceaa7dab`
Signed-off-by: Eric Fried <efried@redhat.com>
* use unstructured to marshal selective fields
Signed-off-by: Alay Patel <alay1431@gmail.com>
* add a sample test for string port in applied config
Signed-off-by: Alay Patel <alay1431@gmail.com>
* update changelog
Signed-off-by: Alay Patel <alay1431@gmail.com>
This change is incompatible with velero-plugin-for-csi
releases <= v0.1.2
Remove special casing of CSI volumesnapshot artifacts
from backup deletion logic as this has been moved to
a DeleteItemAction plugin in the velero-plugin-for-csi repo
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
Xun Jiang
Wenkai Yin(尹文开)
qiuming
Xun Jiang/Bruce Jiang
I538157
yuvalman
Daniel Jiang
half-life666
Scott Seago
Tiger Kaovilai
Tobias Giese
rick
Hoang, Phuong
MatthieuFin
James Landrein
James Landrein
bynare
F. Gold
Dominic Brekau
dkeven
Box-Cube
Frangipani Gold
David L. Smith-Uchida
jxun
Bridget McErlean
Alay Patel
Dharma Bellamkonda
Marcel Haupt
Lars Lehtonen
JenTing Hsiao
Arush Salil
Carlisia Thompson
Kyle William
Eric Fried
Ashish Amarnath