Snapshot tests can be run with Ginkgo focus "Snapshot" and restic tests with Ginkgo focus "Restic".
Restic and volume snapshot tests can now be run simultaneously.
Added check for kibishii app start after restore.
Consolidated kibishii pod checks into waitForKibishiiPods.
Added WaitForPods function to e2e/tests/common.goSnapshot tests are skipped automatically on kind clusters.
Fixed issue where velero_utils InstallVeleroServer was looking for the Restic daemon set in the "velero" namespace only (was ignoring io.Namespace)
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
* Update upgrade docs
Signed-off-by: Carlisia <carlisia@vmware.com>
* Update TOC
Signed-off-by: Carlisia <carlisia@vmware.com>
* The right next version is v1.6.0-beta.1
Signed-off-by: Carlisia <carlisia@vmware.com>
* Correct the listing order
Signed-off-by: Carlisia <carlisia@vmware.com>
* Improve readbility and formatting of pkg/restore/restore.go
Signed-off-by: F. Gold <fgold@vmware.com>
* Update paths to include API group versions
Signed-off-by: F. Gold <fgold@vmware.com>
* Use full word, 'resource' instead of 'resrc'
Signed-off-by: F. Gold <fgold@vmware.com>
The test for multiple credentials assumed that the plugin for the
additional BSL provider was already installed. This will not be the case
when performing a clean install of Velero between tests.
This adds a new utility function to add the plugins that are necessary
for the additional BSL provider. It doesn't check which plugins are
already installed, it will just attempt to install and if the stderr
contains the message that it is a duplicate plugin, we ignore the error
and continue. This could be improved by instpecting the output from
`velero plugin get` but I opted for a quicker solution given the
upcoming release.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Use Credential from BSL for restic commands
This change introduces support for restic to make use of per-BSL
credentials. It makes use of the `credentials.FileStore` introduced in
PR #3442 to write the BSL credentials to disk. To support per-BSL
credentials for restic, the environment for the restic commands needs to
be modified for each provider to ensure that the credentials are
provided via the correct provider specific environment variables.
This change introduces a new function `restic.CmdEnv` to check the BSL
provider and create the correct mapping of environment variables for
each provider.
Previously, AWS and GCP could rely on the environment variables in the
Velero deployments to obtain the credentials file, but now these
environment variables need to be set with the path to the serialized
credentials file if a credential is set on the BSL.
For Azure, the credentials file in the environment was loaded and parsed
to set the environment variables for restic. Now, we check if the BSL
has a credential, and if it does, load and parse that file instead.
This change also introduces a few other small improvements. Now that we
are fetching the BSL to check for the `Credential` field, we can use the
BSL directly to get the `CACert` which means that we can remove the
`GetCACert` function. Also, now that we have a way to serialize secrets
to disk, we can use the `credentials.FileStore` to get a temp file for
the restic repo password and remove the `restic.TempCredentialsFile`
function.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Add documentation for per-BSL credentials
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address review feedback
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address review comments
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
We are no longer adding the Credentials field to the VSL so this reverts
part the change that added it (#3409).
The original PR also added the `snapshot-location set` command. This
command only included options for setting the credential but is part of
the work for #2426. Due to this, the command has been left in place
(with the credentials option removed) but has been hidden.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
This change adds an E2E test which exercises the mulitple credentials
feature added in #3489. The test creates a secret from the given
credentials and creates a BackupStorageLocation which uses those
credentials. A backup and restore is then performed to the default
BSL and to the newly created BSL.
This change adds new flags to the E2E test suite to configure the BSL
created and used in the test.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Load credentials and pass to ObjectStorage plugins
Update NewObjectBackupStore to take a CredentialsGetter which can be
used to get the credentials for a BackupStorageLocation if it has been
configured with a Credential. If the BSL has a credential, use that
SecretKeySelector to fetch the secret, write the contents to a temp file
and then pass that file through to the plugin via the config map using
the key `credentialsFile`. This relies on the plugin being able to use
this new config field.
This does not yet handle VolumeSnapshotLocations or ResticRepositories.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address code reviews
Add godocs and comments.
Improve formatting and test names.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address code reviews
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Add uninstall cmd
- init fn to uninstall velero
- abstract dynamic client creation to a separate fn
- creates a separate client per unstructured resource
- add delete client for CRDs
- export appendUnstructured
- add uninstall command to main cmd
- export `podTemplateOption`
- uninstall resources in the reverse order of installation
- fallback to `velero` if no ns is provided during uninstall
- skip deletion if the resource doesn't exist
- handle resource not found error
- match log formatting with cli install logs
- add Delete fn to fake client
- fix import order
- add changelog
- add comment doc for CreateClient fn
Signed-off-by: Suraj Banakar <suraj@infracloud.io>
* Re-use uninstall code from test suite
- move helper functions out of test suite
- this is to prevent cyclic imports
- move uninstall helpers to uninstall cmd
- call them from test suite
- revert export of variables/fns from install code
- because not required anymore
Signed-off-by: Suraj Banakar <suraj@infracloud.io>
* Revert `PodTemplateOption` -> `podTemplateOption`
Signed-off-by: Suraj Banakar <suraj@infracloud.io>
* Use uninstall helper under VeleroUninstall
- as a wrapper
- fix import related errors in test suite
Signed-off-by: Suraj Banakar <suraj@infracloud.io>
* Validate CRDs against latest Kubernetes versions
Add Kubernetes v1.19 and v1.20 series images, and consolidate the job
into a single file to reduce repetition.
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Ignore job if the changes are only site/design
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Fix codespell error
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Cache Velero binary for reuse on workers
This will cache the Velero binary based on the PR number and a SHA256 of
the generated binary.
This way, the runners testing each version of Kubernetes do not need to
build it independently.
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Fix GitHub event access
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Wrap output path in quotes
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Move code checkout to build step
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Also cache go modules
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Fix syntax issues
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Download cached binary on each node
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
* Use cached go modules on main CI
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
HTTPS requests were failing due to the ca-certificates package not being
installed in the Tilt image.
This change takes the command to install this package from our main
Dockerfile (which also includes installing tzdata).
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Use pod namespace from backup when matching PVBs
In #3051, we introduced an additional check to ensure that a PVB matched
a particular pod by checking both the name and the namespace of the pod.
This caused an issue when using a namespace mapping on restore. In the
case where a namespace mapping is being used, the check for whether a
PVB matches a particular pod will fail as the PVB was created for the
original pod namespace and is not aware of the new namespace mapping
being used. This resulted in PVRs not being created for pods that were
being restored into new namespaces. The restic init containers were
being created to wait on the volume restore, however this would cause
the restored pods to block indefinitely as they would be waiting for a
volume restore that was not scheduled.
To fix this, we use the original namespace of the pod from the backup to
match the PVB to the pod being restored, not the new namespace where
the pod is being restored into.
Fixes#3467.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Explain why the namespace mapping can't be used
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
Split plug-in provider into cloud provider/object provider
Moved velero install/uninstall for tests into velero_utils
Added remove of CRDs to test v elero uninstall
Added remove of cluster role binding to test velero uninstall
Added dump of velero describe and logs on error
Added velero namespace argument to velero_utils functions
Modified api group versions e2e tests to use VeleroInstall
Added velero logs dumps for api group versions e2e testing
Added DeleteNamespace to test/e2e/common.go
Fixed VeleroInstall to use the image specified
Changed enable_api_group_versions_test to use veleroNamespace instead of hardcoded "velero"
Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
* Use kubebuilder client for fetching restic secrets
Instead of using a SecretInformer for fetching secrets for restic, use
the cached client provided by the controller-runtime manager.
In order to use this client, the scheme for Secrets must be added to the
scheme used by the manager so this is added when creating the manager in
both the velero and restic servers.
This change also refactors some of the tests to add a shared utility for
creating a fake controller-runtime client which is now used among all
tests which use that client. This has been added to ensure that all
tests use the same client with the same scheme.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Add builder for SecretKeySelector
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Restore API group version by priority
Signed-off-by: F. Gold <fgold@vmware.com>
* Add changelog
Signed-off-by: F. Gold <fgold@vmware.com>
* Correct spelling
Signed-off-by: F. Gold <fgold@vmware.com>
* Refactor userResourceGroupVersionPriorities(...) to accept config map, adjust unit test
Signed-off-by: F. Gold <fgold@vmware.com>
* Move some unit tests into e2e
Signed-off-by: F. Gold <fgold@vmware.com>
* Add three e2e tests using Testify Suites
Summary of changes
Makefile - add testify e2e test target
go.sum - changed with go mod tidy
pkg/install/install.go - increased polling timeout
test/e2e/restore_priority_group_test.go - deleted
test/e2e/restore_test.go - deleted
test/e2e/velero_utils.go - made restic optional in velero install
test/e2e_testify/Makefile - makefile for testify e2e tests
test/e2e_testify/README.md - example command for running tests
test/e2e_testify/common_test.go - helper functions
test/e2e_testify/e2e_suite_test.go - prepare for tests and run
test/e2e_testify/restore_priority_apigv_test.go - test cases
Signed-off-by: F. Gold <fgold@vmware.com>
* Make changes per @nrb code review
Signed-off-by: F. Gold <fgold@vmware.com>
* Wait for pods in e2e tests
Signed-off-by: F. Gold <fgold@vmware.com>
* Remove testify suites e2e scaffolding moved to PR #3354
Signed-off-by: F. Gold <fgold@vmware.com>
* Make changes per @brito-rafa and Velero maintainers code reviews
- Made changes suggested by @brito-rafa in GitHub.
- We had a code review meeting with @carlisia, @dsu-igeek, @zubron, and @nrb
- and changes were made based on their suggetions:
- pull in logic from 'meetsAPIGVResotreReqs()' to restore.go.
- add TODO to remove APIGroupVersionFeatureFlag check
- have feature flag and backup version format checks in separate `if` statements.
- rename variables to be sourceGVs, targetGVs, and userGVs.
Signed-off-by: F. Gold <fgold@vmware.com>
* Convert Testify Suites e2e tests to existing Ginkgo framework
Signed-off-by: F. Gold <fgold@vmware.com>
* Made changes per @zubron PR review
Signed-off-by: F. Gold <fgold@vmware.com>
* Run go mod tidy after resolving go.sum merge conflict
Signed-off-by: F. Gold <fgold@vmware.com>
* Add feature documentation to velero.io site
Signed-off-by: F. Gold <fgold@vmware.com>
* Add config map e2e test; rename e2e test file and name
Signed-off-by: F. Gold <fgold@vmware.com>
* Update go.{mod,sum} files
Signed-off-by: F. Gold <fgold@vmware.com>
* Move CRDs and CRs to testdata folder
Signed-off-by: F. Gold <fgold@vmware.com>
* Fix typos in cert-manager to pass codespell CICD check
Signed-off-by: F. Gold <fgold@vmware.com>
* Make changes per @nrb code review round 2
- make checkAndReadDir function private
- add info level messages when priorties 1-3 API group versions can not be used
Signed-off-by: F. Gold <fgold@vmware.com>
* Make user config map rules less strict
Signed-off-by: F. Gold <fgold@vmware.com>
* Update e2e test image version in example
Signed-off-by: F. Gold <fgold@vmware.com>
* Update case A music-system controller code
Signed-off-by: F. Gold <fgold@vmware.com>
* Documentation updates
Signed-off-by: F. Gold <fgold@vmware.com>
* Update migration case documentation
Signed-off-by: F. Gold <fgold@vmware.com>
* Use label to select Velero deployment in plugin cmd
Signed-off-by: F. Gold <fgold@vmware.com>
* Move veleroLabel constant closer to usage
Signed-off-by: F. Gold <fgold@vmware.com>
* Add changelog
Signed-off-by: F. Gold <fgold@vmware.com>
* Remove year from copyright in new file
Signed-off-by: F. Gold <fgold@vmware.com>
* Export and use install.Labels() function
Signed-off-by: F. Gold <fgold@vmware.com>
Restoring CAPI workload clusters without this ordering caused the
capi-controller-manager code to panic, resulting in an unhealthy cluster
state.
This can be worked around
(https://community.pivotal.io/s/article/5000e00001pJyN41611954332537?language=en_US),
but we provide the inclusion of these resources as a default in order to
provide a better out-of-the-box experience.
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
The labeler action was failing as it was looking for
`.github/labels.yaml` but the file has the suffix `.yml`. This change
fixes the path used by the labeler action.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
Jalaja
David L. Smith-Uchida
Carlisia Thompson
codegold79
Bridget McErlean
Ashish Amarnath
Nolan Brubaker
Pranav Gaikwad
Suraj Banakar
Madhav Jivrajani
Zadkiel
JenTing Hsiao