Commit Graph

192 Commits (7abd2c6db92ee1c2fde2b21a5080641361ddff2f)

Author SHA1 Message Date
Ashish Amarnath b0fd3d35c1 rename field
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 15:26:44 -07:00
Ashish Amarnath b71173228a add velero server flag to allow default restic use on all pod volumes
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 14:46:06 -07:00
Ashish Amarnath 8a2a852b87 use backup's defaultRestic flag to identify pod volumes using restic
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 14:46:06 -07:00
Nolan Brubaker e400be9c8f
Include CSI volume snapshot information in `velero backup describe` (#2448)
* Add download methods for CSI objects

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add support for downloading CSI volume objects

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add new methods to mock.

Remove generated information from file since mockery no longer appears
to work. It isn't maintained anymore and doesn't support go module-based
projects.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add describe command for CSI

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add csi package with helpers

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove duplicate import from server

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove CSI API that will not be used with describe

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add VolumeSnapshotContents output to describe command

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Document NewCSIListOptions function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Document csi package

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove stutter in function name

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix CI

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix nil pointer error when not using CSI snapshots

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove unused CSI download request kinds

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add back mocks

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Change persistent volumes to velero-native snapshots

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove unused function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Address review feedback

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add changelog

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove unnecessary doc.go

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-08 12:42:30 -07:00
Nolan Brubaker abae81ddc8
Merge pull request #2440 from skriss/backup-progress
report backup progress
2020-05-07 14:51:20 -04:00
Ashish Amarnath 577e87d1b8
Sync CSI API objects during backup sync (#2496)
* add changelog

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* Sync CSI API volumesnapshotcontents during backup sync

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-06 08:00:05 -06:00
Nolan Brubaker 8671a639c9
Backup entire v1beta1 CRD instead of just changing version string (#2478)
* Switch to backing up v1beta1 CRDs from API server

Instead of simply switching out the APIVersion string on a v1
CustomResourceDefinition object, re-download the object from the API
server entirely to get the correct fields.

This should fix validation errors upon restore.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix existing tests

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add full example CRDs to automated tests

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Move beta CRD lookup into helper function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add case for preserveUnknownFields CRDs

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add PreserveUnknownFields case and refactor execute

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add older prometheus CRD test cases

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add changelog

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-05 13:21:28 -07:00
Ashish Amarnath 6b5a084f32
Delete dangling volumesnapshotcontents from velero backups (#2480)
* Delete dangling volumesnapshotcontents from velero backups

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* add changelog

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* handle not found errors from VSC delete

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* clean up unit tests

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-04 15:41:27 -04:00
Steve Kriss 4b0f654a1e add progress info to backup status
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-05-01 15:30:50 -06:00
Ashish Amarnath bf52caf658
Delete CSI volumesnapshots created as part of a backup in its deletion (#2411)
* delete CSI volumesnapshots in backup on backup delete

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* set vsc deletionpolicy to delete prior to vs deletion

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* add more unit tests

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* add change log

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* passes build need to pass unit tests

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* remove GetNamespacesInBackup

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* fix unit tests

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* check feature flag instead of nil check

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* cleanup

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-01 16:05:20 -04:00
Nolan Brubaker d31951c81c Add CSI feature flag constant
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:18 -04:00
Nolan Brubaker ad18318bb1 Rename snapshotter factory variable, return nil in wrapper
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:18 -04:00
Nolan Brubaker 6c3fc8c780 Addresss simple review feedback
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:18 -04:00
Nolan Brubaker 5202885349 Remove unused CSI client variable
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:18 -04:00
Nolan Brubaker 9719e4de9d Don't defer cancelFunc, since it causes issues
Infomers won't start if cancelFunc is invoked as soon as the newServer
function exits via the defer

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:18 -04:00
Nolan Brubaker b7142465fc Wrap usage of the CSI informer in flag checks
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:18 -04:00
Nolan Brubaker 5de15c450b Address some review feedback on server
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:17 -04:00
Nolan Brubaker 6a7beaf5ce Plumb CSI listers through to backup controller
Account for having CSI enabled or not, as well as having the snapshot
CRDs installed in the kubernetes cluster.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:17 -04:00
Nolan Brubaker 232e1d8927 Add clients for CSI snapshots
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-04-20 13:49:17 -04:00
Mayank 016868ecd3
add restore item action to update PVC selected-node annotation (#2377)
* New RestoreItemAction for PVC with annotation `volume.kubernetes.io/selected-node`

Signed-off-by: mayank <mayank.patel@mayadata.io>
2020-04-03 08:01:34 -07:00
Rafael Brito f6cd53c6cb
Adding k8s version labels on backup CRD #2342 (#2346)
Signed-off-by: Rafael Brito <rbrito@vmware.com>
2020-03-19 09:18:05 -07:00
Ashish Amarnath b7cbc4873e add csi types into restore priorities
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-03-06 10:37:03 -08:00
Steve Kriss c7f283c7fa wait for informer caches to sync before running controllers
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-03-03 15:55:17 -07:00
Nolan Brubaker 766ab5d3ee
Account for possible missing schemas on v1 CRDs (#2264)
* Account for possible missing schemas on v1 CRDs

If a v1beta1 CRD without a Schema was submitted to a Kubernets v1.16
cluster, then Kubernetes will server it back as a v1 CRD without a
schema.

However, when Velero tries to restore this document, the request will be
rejected as a v1 CRD must have a schema.

This commit has some defensive coding on the restore side, as well as
potential fixes on the backup side for getting around this.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Back up nonstructural CRDs as v1beta1

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add tests for remapping plugin

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add builders for v1 CRDs

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Address review feedback

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove extraneous log message

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add changelog

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-02-24 12:07:50 -07:00
Nolan Brubaker 6745979a7b
Wait for CRDs to be ready before restoring CRs (#1937)
* Wait for CRDs to be available and ready

When restoring CRDs, we should wait for the definition to be ready and
available before moving on to restoring specific CRs.

While the CRDs are often ready by the time we get to restoring a CR,
there is a race condition where the CRD isn't ready.

This change waits on each CRD at restore time.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-01-30 09:19:13 -08:00
Nolan Brubaker 5b1280c2cd Fix restore issues for CRD v1 on Kubernetes v1.16 clusters (#2197)
Fix restore issues for CRD v1 on Kubernetes v1.16 clusters

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-01-27 13:59:08 -08:00
Steve Kriss be140985c5 bug fix: only prioritize restoring replicasets.apps (#2157)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-12-19 09:44:08 -08:00
Steve Kriss 96297ea437 add replicasets.apps to prioritized restore resources (#2120)
* add replicasets.apps to prioritized restore resources

Signed-off-by: Steve Kriss <krisss@vmware.com>

* changelog

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-12-11 15:32:33 -05:00
Antony S Bett 121dc02d2f Ensure object store plugin processes are cleaned up after restore and after BSL validation during server start up (#2041)
* Ensure child object store plugin processes are cleaned up during restore

Signed-off-by: Antony Bett <antony.bett@gmail.com>
2019-11-05 08:58:08 -08:00
Antony S Bett 0450567bba Allow backup storage locations to specify backup sync period or toggle off sync (#1936)
* Allow backup storage locations to specify backup sync period or toggle off sync

Signed-off-by: Antony Bett <antony.bett@gmail.com>

* Add --backup-sync-period flag to backup location create command

Signed-off-by: Antony Bett <antony.bett@gmail.com>
2019-10-24 08:32:45 -06:00
KubeKween d26bf05b33 Extract providers (#1985)
* Remove cloud providers and reorg code

Signed-off-by: Carlisia <carlisia@vmware.com>

* Update dependencies

Signed-off-by: Carlisia <carlisia@vmware.com>

* Fix tests

Signed-off-by: Carlisia <carlisia@vmware.com>

* fix dependency issues

Signed-off-by: Carlisia <carlisia@vmware.com>

* Delete dup test

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add back spaces to file

Signed-off-by: Carlisia <carlisia@vmware.com>

* Remove and update docs

Signed-off-by: Carlisia <carlisia@vmware.com>

* Make the plugins flag required

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add changelog

Signed-off-by: Carlisia <carlisia@vmware.com>

* Make the plugins flag conditional

Signed-off-by: Carlisia <carlisia@vmware.com>
2019-10-22 15:31:27 -07:00
Alexander Demichev 1564317cef Restore action for cluster/namespace role bindings (#1974)
Signed-off-by: Alexander Demichev <ademicev@redhat.com>
2019-10-21 15:11:26 -06:00
Steve Kriss 48792ece1f Updates for org move to vmware-tanzu (#1920)
* update import paths to github.com/vmware-tanzu/...

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update other GH org refs to vmware-tanzu

Signed-off-by: Steve Kriss <krisss@vmware.com>

* site and docs: update GH org to vmware-tanzu

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update travis badge links on docs readmes

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-09-30 17:26:56 -04:00
Nolan Brubaker 8ec1548b3c Add features package (#1849)
* Add features package

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2019-09-18 10:57:04 -06:00
Steve Kriss 6623e1f273 Change restic prune default interval to 7d and make user-configurable (#1864)
* change restic prune default interval to 7d, add server flag for it

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-09-10 11:58:42 -07:00
Nolan Brubaker 363c2692a1 Use custom namespace when creating Velero clients
The Velero deployment did not have a way of exposing the namespace it
was installed in to the API client. This is a problem for plugins that
need to query for resources in that namespaces, such as the restic
restore process that needs to find PodVolume(Backup|Restore)s.

While the Velero client is consulted for a configured namespace, this
cannot be set in the server pod since there is no valid home directory
in which to place it.

This change provides the namespace to the deployment via the downward
API, and updates the API client factory to use the VELERO_NAMESPACE
before looking at the config file, so that any plugins using the client
will look at the appropriate namespace.

Fixes #1743

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2019-08-13 13:06:12 -04:00
Steve Kriss ffa3251efc move contents of pkg/util/test into pkg/test
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-08-07 08:26:00 -06:00
KubeKween 4accb8512a Restore from PodVolumeBackups (#1723)
* Restore from PodVolumeBackups

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Partially address code reviews

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Partially address code reviews #2

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Clean up struct

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Fix log messages

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Fix tests

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Clean up

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Add changelog

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-08-06 13:17:36 -07:00
carthewd 22eca22ac8 Add JSON as an option for log output (#1654)
* Add JSON as an option for log output

Signed-off-by: Donovan Carthew <donovan.carthew@gmail.com>
2019-07-30 16:29:34 -07:00
KubeKween 3b9af8c654 Store PodVolumeBackups in obj storage & use as source of truth (#1577)
* Store PodVolumeBackups in object storage

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-07-24 15:51:20 -04:00
Steve Kriss 1bb167ef90 add restore item action to change PVC/PV storage class name (#1621)
* add restore item action to change PVC/PV storage class name

Signed-off-by: Steve Kriss <krisss@vmware.com>

* code review

Signed-off-by: Steve Kriss <krisss@vmware.com>

* change existing plugin names to lowercase/hyphenated

Signed-off-by: Steve Kriss <krisss@vmware.com>

* add validation for new storage class name

Signed-off-by: Steve Kriss <krisss@vmware.com>

* add test cases

Signed-off-by: Steve Kriss <krisss@vmware.com>

* changelog

Signed-off-by: Steve Kriss <krisss@vmware.com>

* fix imports

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update plugin names to be more consistent

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update unit tests to use pkg/test object constructors

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-15 11:19:38 -07:00
Adnan Abdulhussein 74b575200c hide server commands in velero CLI (#1561)
* hide server commands in velero CLI

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* changelog

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* remove unnecessary comment

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>
2019-06-10 19:20:17 -06:00
KubeKween 0a771e6a53 Add cmd to list plugins (#1535)
* Add cmd to list plugins

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-06-05 13:41:02 -04:00
Steve Kriss 411d44a673 allow individual backup storage locations to be read-only (#1517)
* allow individual backup storage locations to be read-only

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-05-29 14:21:25 -04:00
Steve Kriss 2406994740 remove ref to v0.10 from BSL validation error msg (#1429)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-05-02 11:20:50 -07:00
Steve Kriss 7f36f78aee remove code that removes legacy GC finalizer from backups
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-15 10:17:03 -06:00
KubeKween abee09aa2d Add validation for plugin name format and dups (#1339)
* Add validation for plugin name format and dups

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Bug fix

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Add changelog

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Address code reviews

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Fix code

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Address code reviews

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Add documentation

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Update godoc

Signed-off-by: Carlisia <carlisiac@vmware.com>

* More doc

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-04-12 08:25:04 -06:00
Aman Wangde 0e0f357cef Added ability to disable controllers (#1326)
Signed-off-by: James King <james.king@emc.com>
2019-04-10 15:41:28 -04:00
Rohan Vora 4beb8aab3c Set default backup TTL
Set default backup TTL to 30 days when TTL
is not provided in the backup yaml configuration.

Updates #138

Signed-off-by: Rohan Vora <vorar@vmware.com>
2019-04-09 14:13:29 -07:00
Fábio Franco Uechi 0167539a14 add new counter metrics for backup deletion (#1280)
* compute backup deletion metrics (attempt, success, fail)

Signed-off-by: fabito <fuechi@ciandt.com>
2019-04-04 14:25:59 -04:00