Commit Graph

940 Commits (e91dfe3d91bc69ede7e739020aa256977ea3f74d)

Author SHA1 Message Date
Steve Kriss 6c9e1f187f lower some noisy logs to debug level
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-11-02 15:55:47 -06:00
Shubheksha Jalan defb8aa856 remove code that checks directly for a backup from restore controller
Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-11-02 23:35:04 +05:30
Michal Wieczorek 7abe115674 Move clearing up of metadata before plugin's actions
Signed-off-by: Michal Wieczorek <wieczorek-michal@wp.pl>
2018-11-02 09:56:33 +01:00
Andy Goldstein cb0e6f4773
Merge pull request #999 from bashofmann/fix-backup-file-deletion-order
AWS: Ensure that the order returned by ListObjects is consistent
2018-10-30 15:46:33 -04:00
Bastian Hofmann 8bbfc538f1 AWS: Ensure that the order returned by ListObjects is consistent
When a backup is deleted, the delete method uses ListObjects to get a list of
files it needs to delete in s3. Different s3 implementations may return
the object lists in different, even non-deterministic orders, which can
result in the deletion not working because ark tries to delete a non empty folder
before it tries to delete the files in the folder.

Signed-off-by: Bastian Hofmann <bashofmann@gmail.com>
2018-10-30 20:38:47 +01:00
Steve Kriss c3c2ea2805
Merge pull request #985 from shubheksha/fix/977-add-prom-metrics-volume-snapshots
Add prometheus metrics for successful and attempted volume snapshots per backup
2018-10-30 13:25:00 -06:00
Shubheksha Jalan 4239e61f47 add prometheus metrics for successful and attempted volume snapshots per backup
rename variables to reflect the metric name

fix comments for exported methods

explicitly record per schedule per schedule metric values

initialize metrics and change variable name to match with that of metric

add metric for recording failed volume snapshots

use singular variable instead of plural

remove extra field for failed snapshots, calculate using existing fields

initialize failure metric and rename methods

Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-10-31 00:47:05 +05:30
Steve Kriss 9165d514a3
Merge pull request #1002 from nrb/fix-609
Verify PV doesn't exist before creating new volume
2018-10-30 13:04:15 -06:00
Nolan Brubaker 6231aaa875 Verify PV doesn't exist before creating new volume
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-10-30 13:57:00 -04:00
Steve Kriss b818cc2769 don't require a default provider VSL if there's only 1
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-26 12:45:40 -06:00
Nolan Brubaker 74cb6a2150
Merge pull request #986 from skriss/fix-empty-bsls
sync controller: fill in missing .spec.storageLocation
2018-10-24 18:38:00 -04:00
Steve Kriss 516422c2c4 sync controller: fill in missing .spec.storageLocation
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-23 16:26:07 -06:00
Steve Kriss 195e6aaf00 fix bug preventing PV snapshots from v0.10 backups from restoring
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-23 12:44:05 -06:00
Andy Goldstein bca585162f Run 'make update' to update formatting
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-10-23 13:08:42 -04:00
Nolan Brubaker 6cf3519c3a
Merge pull request #974 from skriss/backup-describer
backup describer: show snapshot summary by default, details optionally
2018-10-23 12:58:54 -04:00
Nolan Brubaker 6591b5a09c
Merge pull request #975 from skriss/rm-pvproviderexists
remove pvProviderExists param from NewRestoreController
2018-10-23 12:58:17 -04:00
KubeKween 7c4ac05ae5
Merge pull request #972 from skriss/support-pre-v0.10-snaps
Support pre v0.10 snaps
2018-10-23 09:08:52 -07:00
Steve Kriss 90d9be59d3 support restoring/deleting legacy backups with .status.volumeBackups
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-23 09:55:40 -06:00
Steve Kriss f3a57b5b8a
Merge pull request #971 from shubheksha/fix/594-restore-storageclasses-before-pv-pvc
Restore storageclasses before pvs and pvcs
2018-10-23 08:14:42 -06:00
u5surf ef19497205 rename variables #967
Signed-off-by: u5surf <u5.horie@gmail.com>
2018-10-23 08:53:35 +09:00
Shubheksha Jalan 596eea1b8c restore storageclasses before pvs and pvcs
Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-10-22 21:15:08 +02:00
Steve Kriss f014cab1fe backup describer: show snapshot summary by default, details optionally
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-22 12:08:37 -06:00
Steve Kriss 8acc66d02f remove pvProviderExists param from NewRestoreController
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-22 11:47:51 -06:00
u5surf 57ce590fae create a struct for multiple return of same type in restore_contoroller #967
Signed-off-by: u5surf <u5.horie@gmail.com>
2018-10-20 20:45:26 +09:00
Andy Goldstein d0937a3433
Merge pull request #960 from Atul9/return-arguments
#959 Update restoreController runRestore function to specify return arguments
2018-10-19 13:03:30 -04:00
Atul Bhosale db856affcb
Specify return arguments
Update restoreController runRestore function to specify return arguments

Signed-off-by: Atul Bhosale <atul1bhosale@gmail.com>
2018-10-19 21:41:58 +05:30
Domenic Rosati 9952dfb0ad Address #424: Add CRDs to list of prioritized resources
Signed-off-by: Domenic Rosati <domenicrosati@gmail.com>
2018-10-18 22:42:37 -03:00
Wayne Witzel III b3b065a9fb
Merge pull request #949 from skriss/remove-config
remove Config CRD
2018-10-18 10:47:23 -04:00
Steve Kriss cf2c27141b fix bugs in GetBackupVolumeSnapshots and add test
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 15:10:42 -06:00
Steve Kriss c36131a024 remove Config-related code
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 14:00:38 -06:00
Wayne Witzel III 406b50a71b update restore process using snapshot locations
Signed-off-by: Wayne Witzel III <wayne@riotousliving.com>
2018-10-17 13:40:42 -06:00
Steve Kriss 268080ad09 avoid panics if can't get block store during deletion
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:40:42 -06:00
Steve Kriss 4a03370f1d update backup deletion controller for snapshot locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:40:42 -06:00
Steve Kriss 38c72b8cc2 include snapshot locations in created schedule's backup spec
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:40:41 -06:00
Steve Kriss 0ec2de55c0 azure: update blockstore to allow storing snaps in different resource group
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:39:51 -06:00
Steve Kriss 35bb533c2d close gzip writer before uploading volumesnapshots file
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:39:50 -06:00
Steve Kriss da9ed38c63 store volume snapshot info as JSON in backup storage
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:39:49 -06:00
Steve Kriss e24248e07a add --volume-snapshot-locations flag to ark backup create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:36:51 -06:00
Steve Kriss df07b7dc9f update backup code to work with volume snapshot locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:36:50 -06:00
Wayne Witzel III 4af89fa863 add unit test for getDefaultVolumeSnapshotLocations
Signed-off-by: Wayne Witzel III <wayne@riotousliving.com>
2018-10-17 13:35:52 -06:00
Wayne Witzel III 02f50b9c84 add default-volume-snapshot-locations to server cmd
Signed-off-by: Wayne Witzel III <wayne@riotousliving.com>
2018-10-17 13:35:50 -06:00
Carlisia 1aa712d236 Default and validate VolumeSnapshotLocations
Signed-off-by: Carlisia <carlisia@grokkingtech.io>
2018-10-17 13:33:22 -06:00
Wayne Witzel III bbf769850e add create CLI command for snapshot locations
Signed-off-by: Wayne Witzel III <wayne@riotousliving.com>
2018-10-17 13:33:22 -06:00
Wayne Witzel III aeb221eafe Add printer for snapshot locations
Signed-off-by: Wayne Witzel III <wayne@riotousliving.com>
2018-10-17 13:33:22 -06:00
Wayne Witzel III ffc612ac13 Add volume snapshot CLI get command
Signed-off-by: Wayne Witzel III <wayne@riotousliving.com>
2018-10-17 13:33:22 -06:00
Carlisia f20342aab9 Add VolumeLocation and Snapshot.
Signed-off-by: Carlisia <carlisia@grokkingtech.io>
2018-10-17 13:33:22 -06:00
captjt ebc379ac02 Add --include-cluster-resources options when creating schedule
Signed-off-by: captjt <jtaylor007.jt@gmail.com>
2018-10-16 15:52:18 -04:00
KubeKween 0e7f442f00
Merge pull request #930 from skriss/fix-898
don't require a default backup storage location to exist
2018-10-12 12:34:44 -07:00
Wayne Witzel III 37031d5e7c
Merge pull request #920 from skriss/fix-225
only try to backup PVC's linked PV if PVC phase is Bound
2018-10-12 14:10:18 -04:00
Wayne Witzel III 480f1942b7
Merge pull request #927 from skriss/fix-925
pod restore action: check initContainers exist before processing
2018-10-12 14:09:35 -04:00
Steve Kriss 4173515435 don't require a default backup storage location to exist
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-11 14:43:27 -06:00
Andy Goldstein fe51be6713
Merge pull request #926 from shubheksha/fix/329-remove-get-project-gcp
Remove logic to get a gcp project
2018-10-11 15:45:40 -04:00
Steve Kriss 38e86ceff5 pod restore action: check initContainers exist before processing
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-11 13:44:02 -06:00
Shubheksha Jalan 611bc92695 remove logic to get a gcp project
Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-10-11 21:23:30 +02:00
Steve Kriss 63dc6c1d9a only try to backup PVC's linked PV if PVC phase is Bound
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-10 13:34:33 -06:00
KubeKween d579784692
Merge pull request #910 from james-powis/fix_restic_restore_service_user_token
Drop volumes matching name ServiceAccountName-token-
2018-10-09 11:32:23 -07:00
James Powis 30369c2ad5 Drop volumeMounts from initContainers if SAToken
Signed-off-by: James Powis <powisj@gmail.com>
2018-10-05 16:00:58 -06:00
Steve Kriss 0612c5de70 templatize error message in DeleteOptions
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-05 13:24:30 -06:00
Steve Kriss 1da3278ad6
Merge pull request #874 from shubheksha/fix/748-ark-schedule-bulk-deletion
Add support for bulk deletion to `ark schedule delete`
2018-10-05 13:21:40 -06:00
Shubheksha Jalan 66bcbc058c add support for bulk deletion to ark schedule delete
refactor and move DeleteOptions struct and methods

unexport fields not used outside the package in DeleteOptions struct

refactor BindFlags() to work with name of command

fix constructor

Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-10-05 19:45:18 +02:00
Steve Kriss 3af43b492f add azure-specific code to support multi-location restic
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-05 09:20:56 -06:00
Steve Kriss d009163b67 update restic to support multiple backup storage locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-05 09:10:35 -06:00
Wayne Witzel III 1d90d02a99
Merge pull request #900 from skriss/restic-restore-ns-mapping
bug: fix restic restores when using namespace mappings
2018-10-03 10:59:13 -04:00
Steve Kriss 0c88eefc0d bug: fix restic restores when using namespace mappings
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-03 08:09:55 -06:00
Steve Kriss eb709b8f70 only sync a backup location if it's changed since last sync
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-03 08:05:06 -06:00
Steve Kriss 82ab2d73ac
Merge pull request #894 from marctc/fix_backup_list_date
Change CreationTimestamp to StartTimestamp in backup list
2018-10-02 11:04:04 -06:00
Marc Tudurí ae4bf3d5f8 Change CreationTimestamp by StartTimestamp in backup list
Signed-off-by: Marc Tudurí <marctc@gmail.com>
2018-10-02 15:17:58 +02:00
Michal Wieczorek 0f1c5c283f Enable restoring resources with ownerReference set
Signed-off-by: Michal Wieczorek <wieczorek-michal@wp.pl>
2018-10-02 12:04:51 +02:00
KubeKween 94b8fae15a
Merge pull request #891 from skriss/fix-restore-logs
remove restore log helper for accurate line #'s
2018-10-01 21:02:17 -07:00
Steve Kriss 729a688b36 remove restore log helper for accurate line #'s
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-30 14:45:32 -06:00
ffd2subroutine 9fdf85130a Minor code cleanup
Signed-off-by: ffd2subroutine <ffd2subroutine@users.noreply.github.com>
2018-09-30 20:43:11 +02:00
Wayne Witzel III a57acec43d
Merge pull request #876 from skriss/restic-share-bucket
Move restic repos into the main Ark bucket, under restic/
2018-09-28 12:12:45 -04:00
Nolan Brubaker ed2bca83bd
Merge pull request #869 from shubheksha/fix/786-fix-error-ExternalName-service-restore
Fix error when restoring ExternalName services
2018-09-25 16:33:36 -04:00
Steve Kriss e46e89cb61 have restic share main Ark bucket
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-25 13:12:22 -06:00
Steve Kriss 42b54586cd refactor to make valid dirs part of an object store layout
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-25 13:12:22 -06:00
Steve Kriss 8bc7e4f6aa store backups & restores in backups/, restores/ subdirs in obj storage
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-25 09:56:41 -06:00
Shubheksha Jalan e3232b7eb6 add support for bulk deletion to ark restore delete
refactor util methods into a common package

move utility methods into cli package instead of common

rename util.go to common.go

Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-09-25 20:37:20 +05:30
Shubheksha Jalan e3222a9e3f move code dealing with node ports into a separate function
Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-09-25 18:08:57 +05:30
Shubheksha Jalan d95b18bad8 ignore spec.ports not being there for services of type ExternalName
Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-09-25 01:35:13 +05:30
Shubheksha Jalan 907a9fbdd8 fix error during restore when spec.ports are not found
Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>
2018-09-25 01:02:47 +05:30
Steve Kriss 3a60853340
Merge pull request #843 from james-powis/master
Discard all SA Token volumes from pod restore
2018-09-19 13:31:15 -06:00
James Powis a5cb2ce4b7 Change from regex matching default-token to prefix SA-token-
Signed-off-by: James Powis <powisj@gmail.com>
2018-09-19 13:06:10 -06:00
Nolan Brubaker b5e4f85b8c
Merge pull request #834 from skriss/fix-restic-restores
get a new metadata accessor after calling backup item actions
2018-09-12 17:13:16 -04:00
Steve Kriss fcd27a13da get a new metadata accessor after calling backup item actions
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-10 16:19:38 -06:00
Carlisia 69e2f51fbe
Merge pull request #801 from ljakimczuk/master
Claim ownership of new AWS volumes by Kubernetes cluster restoring the backup
2018-09-10 14:03:06 -07:00
Lukasz Jakimczuk ffef86e38d Adding support for the AWS_CLUSTER_NAME env variable allowing to claim volumes ownership
Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>

Moving check for environment variable outside the loop

Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>

Insert a note about AWS_CLUSTER_NAME in the aws-config doc

Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>

Improving implementation and documentation

Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>

Changing instructions, adding unit test for getTagsForCluster and removing duplicated Lookup

Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>

Commit after update

Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>

Correcting bad formatting in aws-config.md

Signed-off-by: Lukasz Jakimczuk <ljakimczuk@gmail.com>
2018-09-10 22:34:33 +02:00
Nolan Brubaker c9a9dd4508
Merge pull request #824 from skriss/generic-controllers
convert all controllers to use genericController, logContext -> log
2018-09-07 14:20:22 -04:00
Steve Kriss 94617b302d convert all controllers to use genericController, logContext -> log
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-07 09:38:58 -06:00
Carlisia 44a75cee7b
Merge pull request #820 from skriss/azure-restic-fix
set azure restic env vars based on default backup location's config
2018-09-06 13:01:45 -07:00
Steve Kriss 7d8813a96c move ObjectStore mock into pkg/cloudprovider/mocks
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-06 10:53:58 -06:00
Steve Kriss f0edf7335f add a BackupStore to pkg/persistence that supports prefixes
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-06 10:53:58 -06:00
Steve Kriss af64069d65 create pkg/persistence and move relevant code from pkg/cloudprovider into it
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-06 09:05:40 -06:00
Steve Kriss 29d75d72e2 move object and block store interfaces to their own files
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-06 09:05:40 -06:00
Zsolt Varga 211aa7b7fd Set schedule labels to subsequent backups
Signed-off-by: Zsolt Varga <zsolt.varga@console.hu>
2018-09-06 10:06:18 +02:00
Steve Kriss d34994cb5f set azure restic env vars based on default backup location's config
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-09-05 14:31:42 -06:00
Steve Kriss cb321db21f azure: refactor to not use helpers/ pkg, validate all env/config inputs
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 9d7ea7483c azure: support different RGs/storage accounts per backup location
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss cd4e9f5336 azure: fix for breaking change in blob.GetSASURI
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss b31e25bf6e server: remove unused code, replace deprecated func
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 729d733986 controllers: take a newPluginManager func in constructors
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 133dc185ca backup sync: process the default location first
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 7a1e6d16cc generic controller: allow controllers with only a resync func
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 6f7bfe545d remove Config CRD's BackupStorageProvider & other obsolete code
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss bd4d97b9e4 move server's defaultBackupLocation into config struct
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 0e94fa37f9 update sync controller for backup locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Carlisia 2750aa71b9 Use backup storage location during restore
Closes #740

Signed-off-by: Carlisia <carlisia@grokkingtech.io>
2018-08-28 13:19:21 -07:00
Steve Kriss 20f89fbcef use the default backup storage location for restic
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Nolan Brubaker 833a6307a9 Add storage location to backup get/describe
Fixes #775

Also conforms ark imports to https://github.com/heptio/ark/issues/494

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss cf7c8587f0 download request: fix setting of log level for plugin manager
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss 3234124afe backup deletion: fix setting of log level in plugin manager
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:20 -07:00
Steve Kriss 74043ab428 download request controller: fix bug in determining expiration
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:20 -07:00
Steve Kriss 7007f198e1 refactor download request controller test and add test cases
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:20 -07:00
Steve Kriss 8f5346150c download request controller: use backup location for object store
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:20 -07:00
Steve Kriss bab08ed1a6 backup deletion controller: use backup location for object store
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:20 -07:00
Nolan Brubaker c6f488f75f Use backup location in the backup controller
Fixes #739

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-28 13:19:20 -07:00
Steve Kriss 06b5af449f add create and get CLI commands for backup locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:19 -07:00
Steve Kriss adbcd3703b add --default-backup-storage-location flag to server cmd
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:14:51 -07:00
Nolan Brubaker 2a34772ed5 Add --storage-location argument to create commands
Closes #738

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-28 13:14:51 -07:00
Nolan Brubaker 56f1617049 Correct metadata for BackupStorageLocationList
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-28 13:14:51 -07:00
Nolan Brubaker 345c3c39b1 Generate clients for BackupStorageLocation
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-28 13:14:51 -07:00
Nolan Brubaker a25eb03290 Add BackupStorageLocation API type
This commit only provides the data model for further work. It does not
implement any logic around locations, nor does it remove anything from
the Config API type.

Closes #736
Closes #732

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-28 13:14:51 -07:00
Nolan Brubaker adc29a2db0
Merge pull request #774 from metadave/dp_report_bug
add an ark bug command
2018-08-24 16:17:21 -04:00
Dave Parfitt 342a1c6437 add an ark bug command
Signed-off-by: Dave Parfitt <diparfitt@gmail.com>
2018-08-24 13:10:56 -04:00
Steve Kriss 8a5d6f9111
Merge pull request #781 from nrb/fix-777
Fix map merging logic
2018-08-22 11:29:02 -07:00
Nolan Brubaker ea50ebf2b5 Fix map merging logic
Fixes #777

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-21 15:30:30 -04:00
Carlisia d29c96387e
Merge pull request #757 from carlisia/c-538-config-m
Switch Config CRD elements to server flags
2018-08-16 09:02:48 -07:00
Carlisia 9508e4a20e
Switch Config CRD elements to server flags
Signed-off-by: Carlisia <carlisia@grokkingtech.io>
2018-08-16 08:51:46 -07:00
Nolan Brubaker 65cd5c602f
Merge pull request #754 from skriss/namespace-restore-label
update how we label restored objects
2018-08-09 15:17:50 -04:00
Steve Kriss 95b2b90006 add backup name label to restored objects
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-08 16:51:33 -07:00
Steve Kriss 0c3ac67b6d start using a namespaced label on restored objects, deprecate old label
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-08 16:34:39 -07:00
Steve Kriss 5acccaa739 add bulk deletion support to ark backup delete
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-08 10:28:40 -07:00
Steve Kriss 1f7a4a1665
Merge pull request #712 from timoreimann/preserve-node-ports-when-specified-in-annotation
Preserve node ports during restore when annotations hold specification.
2018-08-07 09:17:27 -07:00
Timo Reimann 3aa241a74c Preserve node ports during restore when annotations hold specification.
This is to better reflect the intent of the user when node ports are
specified explicitly (as opposed to being assigned by Kubernetes). The
`last-applied-configuration` annotation added by `kubectl apply` is one
such indicator we are now leveraging.

We still default to omitting the node ports when the annotation is
missing.

Signed-off-by: Timo Reimann <ttr314@googlemail.com>
2018-08-07 07:47:17 +02:00
Steve Kriss 27003af62a
Merge pull request #304 from ncdc/wait
Add --wait support to ark backup create
2018-08-03 10:57:20 -07:00
Andy Goldstein c5f5862c9c
Add --wait support to ark backup create
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-08-03 10:42:42 -07:00
Steve Kriss c6ef76d2b3
Merge pull request #719 from nrb/prereq-faq
Document CRD not found errors
2018-08-02 16:03:09 -07:00
Nolan Brubaker eb6f742b5d Document CRD not found errors
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-08-02 16:41:28 -04:00
Carlisia e7bb592602
Add --wait support to `ark restore create`
Signed-off-by: Carlisia <carlisia@grokkingtech.io>
2018-07-31 17:20:03 -07:00
Carlisia 6f061db9a2
Merge pull request #715 from skriss/remove-snapshot-service
remove SnapshotService, replace with direct BlockStore usage
2018-07-31 13:50:22 -07:00
Carlisia 8ce513acbd
Only delete unused backup if they are complete
Fixes #705

Signed-off-by: Carlisia <carlisia@grokkingtech.io>
2018-07-31 09:12:12 -07:00
Steve Kriss 1c26fbde32 remove SnapshotService, replace with direct BlockStore usage
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-31 08:58:11 -07:00
Andy Goldstein 130512187a Refactor plugin management
Refactor plugin management:
- support multiple plugins per executable
- support restarting a plugin process in the event it terminates
- simplify plugin lifecycle management by using separate managers for
  each scope (server vs backup vs restore)

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-07-31 08:34:57 -07:00
Steve Kriss 131afb571e
Merge pull request #607 from nrb/restore-metrics
Restore metrics
2018-07-27 12:25:56 -07:00
Nolan Brubaker 74dbf38793 Add restore failed phase and metrics
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-07-27 14:32:11 -04:00
Steve Kriss ca5656c279 fix bug preventing backup item action item updates from saving
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-26 10:38:17 -07:00
Nolan Brubaker 82f1cd87dc
Merge pull request #680 from carlisia/c-delete-from-etcd
Delete backups from etcd if they're not in storage
2018-07-25 18:25:52 -04:00
Carlisia d2e629f5a8
Delete backups from etcd if they're not in storage
This fixes #309.

Signed-off-by: Carlisia <carlisia@grokkingtech.io>

Co-authored-by: Steve Kriss <steve@heptio.com>
2018-07-25 14:53:50 -07:00
Nolan Brubaker 7cebfe2df0 Add restore attempt and success/failure counters
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-07-25 11:22:43 -04:00
Dave Parfitt 06d6665abb check s3URL scheme upon AWS ObjectStore Init()
Signed-off-by: Dave Parfitt <diparfitt@gmail.com>
2018-07-24 14:01:05 -04:00
Nolan Brubaker b0af81e780
Merge pull request #689 from lemaral/init-schedule-metrics
Init schedule Prometheus metrics
2018-07-23 16:16:26 -04:00
Steve Kriss f62045623c cleanup service account action log statement
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-23 12:10:30 -07:00
Steve Kriss e063b7983a
Merge pull request #682 from nrb/support-rbac-1.7
Support for rbac v1beta1 on kubernetes v1.7.5+
2018-07-23 12:00:50 -07:00
Nolan Brubaker dd1e150511 Add RBAC support for 1.7 clusters
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-07-23 14:19:03 -04:00
Carlisia c47a364ab3
Merge pull request #687 from skriss/fix-delete-issues
fix delete issues
2018-07-23 10:40:14 -07:00
Alex Lemaresquier 450fa72fbb Initialize schedule Prometheus metrics to have them created beforehand (see https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics)
Signed-off-by: Alex Lemaresquier <alex+github@lemaresquier.org>
2018-07-20 23:30:24 +02:00
Steve Kriss 78cbdf95f3 delete old deletion requests for backup when processing a new one
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-20 11:42:42 -07:00
Steve Kriss 85a61b8e8d return nil error if 404 encountered when deleting snapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-20 11:42:34 -07:00
Steve Kriss 1df9a8a38d exit server if not all Ark CRDs exist at startup
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-18 14:30:42 -07:00
Nolan Brubaker e11634bfbc
Merge pull request #676 from skriss/dont-create-namespace
require namespace for backups/etc. to exist at server startup
2018-07-17 14:26:04 -04:00
Steve Kriss 13f893f1f9
Merge pull request #634 from marpaia/k8s-1.11
Update Kubernetes and Client-Go for 1.11.0 / 8.0.0
2018-07-17 08:55:47 -07:00
Mike Arpaia 3efe677007
Update Ark library code to work with Kubernetes 1.11
Signed-off-by: Mike Arpaia <mike@arpaia.co>
2018-07-16 20:46:20 -06:00
Steve Kriss d195512062 require namespace for backups/etc. to exist at server startup
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-13 13:50:12 -07:00
Nolan Brubaker 1e2b141e5d
Merge pull request #443 from skriss/restore-from-schedule
Add --from-schedule flag to `ark restore create`
2018-07-11 13:50:08 -04:00
Steve Kriss 683f7afc0d switch to using .status.startTimestamp for sorting backups
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-11 10:35:55 -07:00
Nolan Brubaker b71a37dbfc Record backup completion time before uploading
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-07-11 13:21:29 -04:00
Nolan Brubaker 09afeb6d3e
Merge pull request #646 from skriss/remove-redundant-filesystem
replace pkg/restore's osFileSystem with pkg/util/filesystem's
2018-07-10 17:57:39 -04:00
Nolan Brubaker fae00a7622
Merge pull request #659 from carlisia/cc-backupsync
Skip backup sync if it already exists on Kubernetes
2018-07-10 12:50:20 -04:00
Carlisia Pinto 5b89f7b6db
Skip backup sync if it already exists in k8s
Signed-off-by: Carlisia Pinto <carlisia@grokkingtech.io>
2018-07-09 21:55:24 -07:00
Steve Kriss c6050845a0 restore controller: switch to 'c' for receiver name
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-09 15:07:38 -07:00
Steve Kriss 706ae07d0d enable a schedule to be provided as the source for a restore
- ScheduleName is added as an API field to the Restore object
- Restore controller validates that exactly one of BackupName
  or ScheduleName has been provided
- If ScheduleName is provided, Restore controller populates
  BackupName with the name of the most recent successful backup
  created from the schedule
- --from-schedule flag is added to `ark restore create` CLI cmd

Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-09 15:07:38 -07:00
Steve Kriss 795dc26214 replace pkg/restore's osFileSystem with pkg/util/filesystem's
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-05 15:46:17 -07:00
Andy Goldstein 2999f158db
Merge pull request #550 from skriss/restic-describers
Add pod volume backups/restores to ark backup/restore describe
2018-07-05 16:27:36 -04:00
Andy Goldstein 8dd9cded1a
Fix testing arbitrary map key order
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-07-05 16:22:46 -04:00
Steve Kriss 42f2891485 add pod volume backups/restores to ark backup/restore describe output
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-05 13:13:40 -07:00
Andy Goldstein 9db5e36b54
Fix test sorting function
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-07-05 16:08:05 -04:00
Steve Kriss c18decc89b pkg/restic unit tests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-05 11:44:17 -07:00
Steve Kriss 547625c333 set --hostname flag for restic backups
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-02 10:52:32 -07:00
Mike Arpaia eabef08561
Update generated Ark code based on the 1.11 k8s.io/code-generator script
Signed-off-by: Mike Arpaia <mike@arpaia.co>
2018-07-02 09:24:19 -06:00
Steve Kriss ae4aad0890 use pkg/util/exec for running get snapshot id cmd
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-29 12:06:31 -07:00
Andy Goldstein fe286ff564 Don't restore backups or restores
Add backups and restores the list of non restorable resources. Backups,
if applicable, are synced from object storage by the backup sync
controller. Restores are specific to a cluster and don't have value
moving across clusters.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-06-28 16:58:09 -04:00
Andy Goldstein eaeb9d677e
Merge pull request #608 from skriss/no-pv-snapshot-if-restic-backup
don't snapshot volumes that have been backed up with restic
2018-06-28 13:27:30 -04:00
Steve Kriss 11c176c490 don't snapshot volumes that have been backed up with restic
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-28 10:19:34 -07:00
Andy Goldstein 539de6d361
Merge pull request #564 from nrb/backup-timing
Record backup start and completion times, add timing metrics
2018-06-28 11:37:39 -04:00
Nolan Brubaker 96b72acb2d Record backup start and completion times
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-06-28 11:18:14 -04:00
Andy Goldstein fa470170cf
Merge pull request #610 from skriss/restic-skip-invalid-volumes
log and skip backups for non-existent & hostPath volumes
2018-06-28 10:49:33 -04:00
Andy Goldstein 255a991c6e Stop restoring mirror pods
Mirror pods are pods created from static manifest files on a node.
They're mirrored to the apiserver so they're visible when querying the
apiserver for a list of pods, but it's not possible to send a pod
containing the mirror pod annotation to the apiserver and have it be
created successfully. Instead of trying to do this, log a message that
we're skipping restoring the pod because it's a mirror pod.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-06-28 10:06:55 -04:00
Steve Kriss 781b7cd1aa log and skip backups for non-existent & hostPath volumes
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-27 14:37:36 -07:00
Steve Kriss 22e8f23e2c replace ark restic repo init cmd with automatic initialization of repos
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-27 13:50:37 -07:00
Steve Kriss a697ad164e refine what gets enqueued in PVB/PVR controllers, and log better
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-27 13:31:11 -07:00
Nolan Brubaker 29ac0b4a6c
Merge pull request #613 from ncdc/ignore-reclaim-policy-delete-pvs-without-snapshots
Skip reclaim policy Delete PVs without snapshots
2018-06-26 16:52:01 -04:00
Andy Goldstein 7c283e5de8 Skip reclaim policy Delete PVs without snapshots
If a PV has a reclaim policy of Delete and we didn't create a snapshot
of it, don't restore the PV, as doing so would create a PV whose
underlying volume is incorrect.

Also "reset" any PVCs bound to the PV so they'll be dynamically
provisioned when restored.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-06-26 10:40:49 -04:00
Nikhita Raghunath 0da5f1ccca Add omitempty tag to optional API fields
Signed-off-by: Nikhita Raghunath <nikitaraghunath@gmail.com>
2018-06-25 16:40:43 +05:30
Steve Kriss 5ad21854f7 restic: if S3, get bucket's region up-front
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-21 17:20:53 -07:00
Steve Kriss 25d3597c9a add a /scratch emptyDir to ark pods to use for restic cache
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 11:48:09 -07:00
Steve Kriss f2072e5868 use repo identifier from ResticRepo CR when backing up/restoring
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 10:42:53 -07:00
Steve Kriss a927906e52 add --maintenance-frequency flag to ark restic repo init
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 09:21:43 -07:00
Steve Kriss 6e9e653f76 add restic repo getter and reorg restic cmds
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 09:21:43 -07:00
Steve Kriss 3481618324 move runCommand to pkg/util/exec and use in restic repo mgr
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 09:21:43 -07:00
Steve Kriss 65ed8da4b7 add ResticRepository CRD and move repo-level actions to a controller
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 09:21:43 -07:00
Steve Kriss de12ca4882 move restic binary into /usr/bin so it's in PATH
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-19 11:31:27 -07:00
Steve Kriss 01b5828ee7
Merge pull request #403 from nrb/merge-serviceaccount-secrets
Merge serviceaccounts on restore
2018-06-15 09:46:26 -07:00
Nolan Brubaker e7d00cf5fd Add merge support for serviceaccounts
All properties from a backup will be merged into the ServiceAccount
except for the default token secret.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-06-15 12:40:59 -04:00
Steve Kriss 2dfa7a1a72
Merge pull request #553 from ncdc/log-to-stdout
Log to stdout instead of stderr
2018-06-15 08:58:08 -07:00
Steve Kriss 6fb11b8087 get rid of restore staging dir by backing up/restoring within volume dir
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-14 13:49:27 -07:00
Andy Goldstein cc9140b3cc Log to stdout instead of stderr
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-06-14 14:02:12 -04:00
Andy Goldstein 7be81fe60e
Merge pull request #546 from skriss/rm-old-donefiles
remove existing .ark dir and contents during restic restores
2018-06-14 11:12:31 -04:00
Ashish Amarnath 05e86ee734 expose container ports in example deployment and pkg/install tooling
Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2018-06-13 19:22:09 -07:00
Steve Kriss dc273e3bed remove existing .ark dir and contents during restic restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-13 16:52:06 -07:00
Andy Goldstein 9245e9d5dc
Merge pull request #547 from skriss/restic-rsync
use rsync to copy restored data from staging dir to volume dir
2018-06-13 16:56:42 -04:00
Steve Kriss f7a42f378f use rsync to copy restored data from staging dir to volume dir
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-13 13:41:55 -07:00
Andy Goldstein d1e3688468
Merge pull request #531 from ashish-amarnath/add-backup-metrics
Add metrics package to publish metrics
2018-06-13 16:33:30 -04:00
Ashish Amarnath 83658e891e Add a metrics package to add and expose metrics
* add a metrics package to handle metric registration and publishing
* add a metricsAddress field to the server struct
* make metrics a part of the server
* start a metrics endpoint as part of starting the controllers
* instrument backup_controller to report metrics
* update cli-reference docs
* update example deployments with prometheus annotations
* update 'pkg/install' tooling with prometheus annotations

Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2018-06-13 13:17:08 -07:00
Steve Kriss d7134b1df2 use Ark version as the tag for the restore-helper image in init container
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 10:46:10 -07:00
Steve Kriss 81520a9b86 rename restic-init-container to ark-restic-restore-helper
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 10:24:23 -07:00
Steve Kriss 2c6fc5bd90 don't fail the restore if cleanup of the staging directory post-restore fails
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:48:26 -07:00
Steve Kriss 6897c2f901 move dotfiles from staging to volume dir on restore
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:31:03 -07:00
Steve Kriss e354b1c130 replace complete-restore.sh with go code
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:31:03 -07:00
Steve Kriss c1f4e6d92d rename ark daemonset to ark restic server
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:12:49 -07:00
Andy Goldstein 0396ca1dee
Merge pull request #526 from yastij/ignore-resources-terminating
ignore terminating resources while doing a backup
2018-06-08 11:10:59 -04:00
Yassine TIJANI 17f6a14d37 ignore terminating resources while doing a backup
Signed-off-by: Yassine TIJANI <yasstij11@gmail.com>
2018-06-08 16:49:35 +02:00
Steve Kriss 67b40c7fc8 use fake filesystem to test key file flag for ark restic init-repo
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-07 10:29:59 -07:00
Steve Kriss e2561f9073 ark restic init-repository: use key data if provided
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-07 10:06:15 -07:00
Steve Kriss 50d4084fac add restic integration for doing pod volume backups/restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-06 09:48:10 -07:00
Andy Goldstein 86b9cc6d15 BackupItemActionPlugin: handle nil updatedItem
Handle the case where a BackupItemAction may return nil for updatedItem,
meaning "no modifications to the item". The backupPVAction does this,
and we were panicking instead of accepting it.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-06-01 14:03:32 -04:00
Calle Pettersson b2ec87f05f Run post-hooks even if backup actions fail
Signed-off-by: Calle Pettersson <cpettsson@gmail.com>
2018-05-23 18:44:55 +02:00
Steve Kriss fb33d93186
Merge pull request #512 from carlpett/additional-backup-error
Handle errors in additionalItemBackupper
2018-05-23 09:02:20 -07:00
Steve Kriss 170034787d
Merge pull request #513 from ncdc/510-follow-ups
Fix copyright date, add happy path test case
2018-05-23 08:54:22 -07:00
Calle Pettersson b92d086712 Add additional test
Signed-off-by: Calle Pettersson <cpettsson@gmail.com>
2018-05-23 17:28:51 +02:00
Andy Goldstein 18e2401e79
Merge pull request #455 from skriss/remove-aws-region
AWS: remove region requirement from config for backupStorageProvider
2018-05-23 11:02:57 -04:00
Andy Goldstein 849297e623 Fix copyright date, add happy path test case
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-05-23 09:59:32 -04:00
Andy Goldstein 92e9d307a5
Merge pull request #510 from nrb/failed-backup-upload-491
Fail backup if object store upload fails
2018-05-23 09:55:11 -04:00
Nolan Brubaker 09bbe072cd Check errors when closing GCP objects
Writing to GCP's object store is any async operation, so errors need to
be checked both on write and close calls, since errors like permission
violations aren't reported until a close.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-22 17:38:49 -04:00
Steve Kriss b029860b46 AWS: remove region requirement from config for backupStorageProvider
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-22 11:59:53 -07:00
Calle Pettersson 24dfef6f15 Handle errors in additionalItemBackupper
Signed-off-by: Calle Pettersson <cpettsson@gmail.com>
2018-05-22 18:32:48 +02:00
Mike Arpaia cb7bcea5c3
Add functional options for the Ark config install library
Signed-off-by: Mike Arpaia <mike@arpaia.co>
2018-05-22 09:21:12 -06:00
Andy Goldstein aeb5f6d832
Merge pull request #437 from marpaia/install-api
Add library code to install required server components
2018-05-18 16:48:42 -04:00
Mike Arpaia 16f707aa11
Add library code to install required server components
This PR includes Go library code to assist with the installation of
required server-side components.

Signed-off-by: Mike Arpaia <mike@arpaia.co>
2018-05-18 14:43:06 -06:00
Andy Goldstein 2fde1f5fc1
Merge pull request #500 from nrb/fix-497
Exit when failing to update a backup's phase
2018-05-17 14:23:41 -04:00
Andy Goldstein 3a746a3f73
Merge pull request #496 from nrb/fix-477
Check restore Phase before downloading logs
2018-05-17 13:23:23 -04:00
Andy Goldstein f288902e3e
Merge pull request #490 from skriss/jsonpatch
Use JSON merge patches everywhere
2018-05-17 13:02:47 -04:00
Nolan Brubaker 7eac6675e8 Exit when failing to update a backup's phase
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-17 12:47:38 -04:00
Nolan Brubaker f6761ddd00 Check restore Phase before downloading logs
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-17 12:20:55 -04:00
Steve Kriss 5d74a92cf1
Merge pull request #484 from nrb/move-groupresources
Consolidate commonly used GroupResource objects
2018-05-15 14:02:29 -07:00
Nolan Brubaker f936c55a37 Consolidate commonly used GroupResource objects
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-15 15:55:15 -04:00
Andy Goldstein ad93135adb
Merge pull request #483 from skriss/graceful-shutdown-on-sigs
shutdown gracefully on SIGINT/SIGTERM
2018-05-15 14:33:09 -04:00
Andy Goldstein 67263d2652
Merge pull request #486 from nrb/fix-475
Filter on resources that support get & delete
2018-05-15 10:58:56 -04:00
Nolan Brubaker 4fcd222777 Expand required resource verbs with get and delete
Some resources use GET for listing, which resulted in errors.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-15 10:55:03 -04:00
Steve Kriss 6d6f734bc9 use json merge patches
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-14 14:34:24 -07:00
Steve Kriss 1c950aa17b unit test to ensure new cohabitatingResources map for each backup
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-14 10:23:21 -07:00
Steve Kriss ed7fbc9178 use new cohabitatingResources map for each backup
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-11 16:30:14 -07:00
Steve Kriss 20f56e9868 cleanup plugin subprocesses on server exit
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-11 09:48:46 -07:00
Steve Kriss 09c20b51e6 shutdown gracefully on SIGINT/SIGTERM
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-10 16:22:14 -07:00
Steve Kriss fbb5ead4e9 add replicasets and daemonsets to cohabitating resources
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-10 14:15:01 -07:00
Andy Goldstein 43b1f9a19e
Merge pull request #418 from skriss/refactor-patch-tests
use typed structs for decoding patch JSON in unit tests
2018-05-09 15:44:21 -04:00
Andy Goldstein ea83ed32f5
Merge pull request #470 from skriss/backup-cluster-roles-and-bindings
backups: include clusterroles/bindings that reference serviceaccounts
2018-05-09 13:21:13 -04:00
Steve Kriss 041cfc2173 backups: include clusterroles/bindings that reference serviceaccounts
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-09 10:03:55 -07:00
Andy Goldstein ef5ac7fd05
Merge pull request #463 from nrb/fix-454
Don't restore completed pods or jobs
2018-05-09 09:59:34 -04:00
Steve Kriss e81de2491f remove duplicate github.com/satori/go.uuid dependency
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 10:02:59 -07:00
Steve Kriss 51928e9177 use typed structs for decoding patch JSON in unit tests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 09:22:20 -07:00
Steve Kriss 6754955bcd when syncing backups, set their namespace to current cluster's Ark ns
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 09:12:51 -07:00
Nolan Brubaker 923870390b Skip completed jobs and pods when restoring
Completed jobs and pods may be useful in the backup for auditing
purposes, but don't recreate them when restoring.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-02 12:16:54 -04:00
Andy Goldstein 18d6b233da
Merge pull request #453 from nrb/fix-444
Make empty excludes string more accurate
2018-04-24 14:53:33 -04:00
Andy Goldstein c8989231eb
Merge pull request #417 from skriss/upgrade-k8s-deps
Upgrade apimachinery, client-go, api, kubernetes dependencies
2018-04-24 14:53:11 -04:00
Nolan Brubaker 05cb059b1a Make empty excludes string more accurate
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-24 14:34:56 -04:00
Steve Kriss 72d7e5e01d azure: pass full snapshot name when restoring disk
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-23 12:51:05 -07:00
Steve Kriss a466208975 add backwards-compatibility for azure snapshot ID
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-23 11:34:37 -07:00
Steve Kriss 3dc093c24a regenerate code with updated generator
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:59:22 -07:00
Steve Kriss 989169dcfe remove unused kubernetes/pkg/util/version dependency
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:59:22 -07:00
Steve Kriss 97e52f2b3c add timeout arg to rest.NewRequest()
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:58:53 -07:00
Steve Kriss 961d7f2924 kubernetes/pkg/printers/ShortHumanDuration -> apimachinery/pkg/util/duration
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:58:53 -07:00
Nolan Brubaker 300a010959 Remove finalizers from backups before syncing
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-18 17:32:39 -04:00
Andy Goldstein 730ddb2113
Merge pull request #391 from containscafeine/handle-restore-missing-error
Introduce validation logic to `ark restore logs`
2018-04-12 11:07:29 -04:00
Shubham Minglani 7f3e88151b Add bash and zsh completion support
This commit adds support for auto completion for bash and zsh
shells. A new root level command called "completion" has been
introduced, and the user can get the auto completion code by
running `ark completion bash/zsh`.

For bash completion, the built-in GenBashCompletion() from cobra
has been used, but for zsh, the built-in GenZshCompletion() is
known to cause issues. The workaround has been copied from zsh
completion code of kubectl.

Signed-off-by: Shubham <shubham@linux.com>
2018-04-12 13:26:23 +05:30
Shubham 468c4faf1b Introduce validation logic to `ark restore logs`
This commit introduces validation logic to `ark restore logs`
command, the way it already exists in other commands like `ark
restore create`.

Before the logs for a restore are fetched from the server, the
server is contacted to check if the specified restore exists. If
it does not, it errors out.

Fixes #389

Signed-off-by: Shubham <shubham@linux.com>
2018-04-12 13:06:32 +05:30
Andy Goldstein 0998f6d8af
Merge pull request #405 from nrb/ignore-duplicates
Compare for duplicates before logging object exists errors
2018-04-11 09:47:37 -04:00
Nolan Brubaker 33cc85cd0c Compare backup and cluster objects before logging
When restoring resources that raise an already exists error, check their
equality before logging a message on the restore. If they're the same
except for some metadata, don't generate a message.

The restore process was modified so that if an object had an empty
namespace string, no namespace key is created on the object. This was to
avoid manipulating the copy of the current cluster's object by adding
the target namespace.

There are some cases right now that are known to not be equal via this
method:

- The `default` ServiceAccount in a namespace will not match, primarily
because of differing default tokens. These will be handled in their own
patch
- IP addresses for Services are recorded in the backup object, but are
either not present on the cluster object, or different. An issue for
this already exists at https://github.com/heptio/ark/issues/354
- Endpoints have differing values for `renewTime`. This may be
insubstantial, but isn't currently handled by the resetMetadataAndStatus
function.
- PersistentVolume objects do not match on spec fields, such as
claimRef and cloud provider persistent disk info

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-10 14:55:48 -04:00
Andy Goldstein 258f3e011e
Merge pull request #409 from skriss/fix-398
get backup name from restore spec when handling restore DownloadRequests
2018-04-10 13:29:07 -04:00
Steve Kriss 0d2b49acce get backup name from restore spec when handling restore DownloadRequests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 10:50:10 -07:00
Steve Kriss 5bb47d2b7a extract a helper for comparing slices of Actions reliably
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 09:39:33 -07:00
Andy Goldstein 3f2e222ae4 Don't allow deletion of in-progress backups
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-06 13:08:39 -04:00
Andy Goldstein 644a75e3c1 Set DeleteBackupRequest labels if missing
When the BackupDeletionController processes a request, set the request's
backup-name and backup-uid labels if they aren't currently set.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 15:38:44 -04:00
Andy Goldstein ef57a44827 Require DeleteBackupRequest.Spec.BackupName
Make sure a DeleteBackupRequest has its Spec.BackupName filled in. If
not, record an error in the status and mark the request as processed.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 14:53:10 -04:00
Andy Goldstein 4328b67f93 backup delete/describe enhancements
Always request DeleteBackupRequests for a given backup so we can show
failed deletion attempts if you try to delete a backup that has PV
snapshots when Ark doesn't have a persistentVolumeProvider configured.

When creating a DeleteBackupRequest, include a label for the UID so we
can match based on name and UID when associated DeleteBackupRequests
with a given backup.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 14:21:45 -04:00
Andy Goldstein 1ae492da6c Expire processed deleted backup requests > 24hr
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 11:19:11 -04:00
Andy Goldstein 74f60b1ee1 Switch backup finalizer to DeleteBackupRequest
We ran into a lot of problems using a finalizer on the backup to allow
the Ark server to clean up all associated backup data when deleting a
backup.

Users also found it less than desirable that deleting the heptio-ark
namespace resulted in all the backup data being deleted.

This removes the finalizer and replaces it with an explicit
DeleteBackupRequest that is created as a means of requesting the
deletion of a backup and all its associated data. This is what `ark
backup delete` does.

If you use kubectl to delete a backup or to delete the heptio-ark
namespace, this no longer deletes associated backups. Additionally, as
long as the heptio-ark namespace still exists, the Ark server's
BackupSyncController will continually sync backups into the heptio-ark
namespace from object storage.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 11:16:15 -04:00
Steve Kriss 80b66434c0 move getting client into Complete()
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-03 20:43:42 -07:00
Steve Kriss c60e47dedd use cobra's arg-count validation & call Complete() before Validate()
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-03 09:45:29 -07:00
Shubham 73499c2cff Add limitranges to defaultResourcePriorities
This commit adds limitranges to defaultResourcePriorities as
suggested in #385.

This is done so that pods are not restored before the LimitRange
objects, because that would lead to pods not honoring the requests
and limits set in LimitRange objects.

Fixes #385

Signed-off-by: Shubham <shubham@linux.com>
2018-03-17 21:13:44 +05:30
Shubham 560f9504e1 Fix typo: replace ; with :
This commit replaces a ; with a : in an error message

Signed-off-by: Shubham <shubham@linux.com>
2018-03-16 18:24:34 +05:30
Andy Goldstein 3172e907a1
Merge pull request #341 from skriss/snapshot-tags
Add tags to snapshots, restored volumes
2018-03-14 12:50:40 -04:00
Andy Goldstein 07fcc927b7
Merge pull request #359 from skriss/reduce-azure-polling-interval
Azure: reduce client polling interval from default 60s to 5s
2018-03-14 12:17:54 -04:00
Steve Kriss 45cee7d222 Azure: reduce client polling interval from default 60s to 5s
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-14 09:07:20 -07:00
Steve Kriss b2b16b3c85 Azure: fix bug preventing snapshot deletion
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 17:03:14 -07:00
Steve Kriss 0388845bca AWS: tag snapshots during create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss a05ae1a7cf add useful Ark tags to snapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss df985bca74 Azure: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss cab904570f GCP: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 11:52:09 -07:00
Steve Kriss 9673e9d158 AWS: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 11:52:09 -07:00
Steve Kriss 51c546786e
Merge pull request #374 from sanketjpatel/feat/exclude-events-restore
Add events to nonRestorableResources and cohabitatingResources
2018-03-13 11:23:17 -07:00
Andy Goldstein f83b1de1dc
Merge pull request #356 from skriss/store-azure-snapshot-uri
Azure: store snapshot URI to support cross-resource group restores
2018-03-13 13:45:48 -04:00
Steve Kriss 409f17361d Azure: store snapshot URI to support cross-resource group restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 09:18:20 -07:00
Sanket Patel 7257a75f82 Add events to nonRestorableResources and cohabitatingResources
Fixes #367
Fixes #368

Signed-off-by: Sanket Patel <sanketpatel.301090@gmail.com>
2018-03-12 19:55:32 -05:00
Andy Goldstein 178a007b70
Merge pull request #362 from skriss/tweak-hook-logging
during backup, only log if hooks are actually executing
2018-03-07 13:37:03 -05:00
Steve Kriss 26944cf9a2 only log if hooks are actually executing
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-07 09:29:27 -08:00
Andy Goldstein 7328fd0e10
Merge pull request #343 from skriss/gc-bug
add an AddFunc to handle finalizers in GC for initial lists/resyncs
2018-03-06 16:41:57 -05:00
Andy Goldstein 331e0c28cc Remove Azure location requirement
Instead of requiring the Ark admin to specify a "location" in the azure
persistentVolumeProvider config (meaning only a single location is
supported), get info about the disk (for its location) when creating a
snapshot, and get info about the snapshot (for its location) when
creating a disk from a snapshot.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-03-05 15:20:58 -05:00
Steve Kriss c281124cba add --from-backup flag to ark restore create & allow restore name
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-05 10:06:43 -08:00
Andy Goldstein 6930c846e5 Fix incorrect return when err != nil
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-03-05 10:57:23 -05:00
Andy Goldstein cc9be4489a
Merge pull request #345 from skriss/gcp-project
extract GCP project ID from creds file, remove from config
2018-03-05 10:35:39 -05:00
Steve Kriss cbff5ef53b extract GCP project ID from creds file, remove from config
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-02 09:15:12 -08:00
Steve Kriss 509431909a add an AddFunc to handle finalizers in GC for initial lists/resyncs
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-02-28 16:56:23 -08:00
Steve Kriss 975dce929c
Merge pull request #331 from lypht/schedule
Include cron examples for schedule creation
2018-02-28 15:32:28 -08:00
Josh Hull f733869142
Include cron examples for schedule creation
Signed-off-by: Josh Hull <josh.hull@lypht.com>
2018-02-28 16:50:52 -06:00
Andy Goldstein e618e0e456 server: allow configurable plugin dir
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-27 14:50:38 -05:00
Andy Goldstein f13b0c00a3 server: don't assume /ark for internal plugins
If you want to test changes to the ark server without having to rebuild
and redeploy the ark container, this change allows you to do something
like this (assuming you've created your cloud credentials file):

AWS_SHARED_CREDENTIALS_FILE=credentials-minio ark server -n heptio-ark

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-27 14:32:45 -05:00
Nolan Brubaker e73ba839f3
Merge pull request #296 from blakebarnett/bdb/add_kubecontext_flag
Adds a kubecontext global flag
2018-02-26 17:05:59 -05:00
Nolan Brubaker abeeee6643
Merge pull request #333 from ncdc/add-force-delete-backup
Allow forced backup deletion
2018-02-26 16:44:37 -05:00
Andy Goldstein d24fb232cc Allow forced backup deletion
Add --force and --confirm to `ark backup delete` to support forced
backup deletion. This forcibly removes the Ark GC finalizer (if it's
present) from a backup and will orphan any resources associated with the
backup, such as backup tarballs in object storage, persistent volume
snapshots, and restores for the backup.

If a backup has a deletion timestamp, display `Deleting` in `ark backup
describe` and `ark backup get`.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-26 16:25:22 -05:00
Blake 465c60b408
Add global kubecontext CLI flag
Signed-off-by: Blake <blake.barnett@postmates.com>
2018-02-26 10:47:39 -08:00
Andy Goldstein cc4d2916fa Error if backup missing when creating a restore
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-05 13:29:48 -05:00
Andy Goldstein b2cd8e1fe8 Create the config directory in case it's missing
Always try to create the config directory when saving the client config
in case it doesn't exist.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-02 16:01:52 -05:00
Andy Goldstein 6712e2a9ec
Merge pull request #270 from nrb/fix-232
Clarify backup and restore creation messages
2018-01-11 12:22:11 -05:00
Andy Goldstein f5f1dbac5d
Merge pull request #271 from nrb/fix-246
Invalidate restores based on non-existent backups
2018-01-10 13:24:53 -05:00
Steve Kriss af81f589f6 update GCP block store to use v1 library, and tag snapshots on create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-08 09:23:32 -08:00
Nolan Brubaker 37f498cd37 Clarify backup and restore creation messages
When running `ark <resource> create`, a request is sent to the server,
but the status is not immediately known. Inform the user that a request
was sent and provide a way to get more information on it.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-01-08 10:26:51 -05:00
Nolan Brubaker bb7b0a3c33 Invalidate restores based on missing backups
When creating a restore based on a backup that doesn't exist, the
restore should be marked as invalid and the error clearly communicated
so the user understands why the restore wasn't made.

Previously, the restore was left as in progress with an error attached.

Since restores are CRDs and must be updated via a controller, there's
currently not a way to give the client immediate errors.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-01-08 10:10:01 -05:00
Steve Kriss b53ee44176 update GCP object store to use new/recommended lib
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-05 14:21:13 -08:00
Steve Kriss e130011d1e
Merge pull request #272 from ncdc/arbitrary-namespaces
Support running in any namespace
2018-01-05 14:15:18 -08:00
Andy Goldstein 816f14c0b4 Support running in any namespace
Add the ability for the Ark server to run in any namespace.

Add `ark client config get/set` for manipulating the new client
configuration file in $HOME/.config/ark/config.json. This holds client
defaults, such as the Ark server's namespace (to avoid having to specify
the --namespace flag all the time).

Add a --namespace flag to all client commands.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-05 16:51:59 -05:00
Andy Goldstein de785af89d Support pre and post hooks.
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-05 10:47:34 -05:00
Andy Goldstein d56b5f2b66 Updated generated CRD code
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-04 10:44:20 -05:00
Steve Kriss 656428d0b9
Merge pull request #250 from ncdc/backup-controller-do-as-much-as-possible
BackupController: do as much as possible
2018-01-03 12:10:51 -08:00
Andy Goldstein 1b124a3146 Log backup item action error when it occurs
If a backup item action errors, log the error as soon as it occurs, so
it's clear when the error happened. Also include information about the
groupResource, namespace, and name of the item in the error.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-03 13:20:05 -05:00
Andy Goldstein 6b0b6373da
Merge pull request #252 from skriss/delete-backup
add delete backup cmd using finalizer and simplify GC process
2018-01-02 14:26:03 -05:00
Andy Goldstein 56ca4bab2f
Merge pull request #268 from skriss/2018
update copyright header to 2018 and make language/casing consistent
2018-01-02 14:16:26 -05:00
Steve Kriss 1503796419 make language and casing consistent across copyrights
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-02 10:51:49 -08:00
Steve Kriss 3b157afbbd update copyright year to 2018 for generated files
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-02 10:48:42 -08:00
Steve Kriss 1c974782fa disable GC and backup deletion if Kubernetes is less than v1.7.5
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-22 10:25:32 -08:00
Steve Kriss 8878ba860e don't remove annotations from PVs on restore
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-21 13:23:48 -08:00
Steve Kriss 8e5feec39c include restore & schedule under ark delete
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-21 10:47:37 -08:00
Steve Kriss f5123794e0 add delete backup cmd using finalizer and simplify GC process
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-21 10:47:37 -08:00
Andy Goldstein 0fc087c967 Flatten aggregated errors
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-12-21 11:56:33 -05:00
Andy Goldstein fbda5fe539 Add serviceaccounts to prioritized resources
Add serviceaccounts to the default list of prioritized resources used
when restoring.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-12-20 14:59:47 -05:00
Andy Goldstein 1e581f1ead BackupController: do as much as possible
When running a backup, try to do as much as possible, collecting errors
along the way, and return an aggregate at the end. This way, if a backup
fails for most reasons, we'll be able to upload the backup log file to
object storage, which wasn't happening before.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-12-20 14:49:27 -05:00
Andy Goldstein fea701a03b
Merge pull request #256 from skriss/unbound-pvc-fix
don't error if PVC is not bound to a PV
2017-12-19 19:37:28 -05:00
Andy Goldstein 1210cb36e1
Merge pull request #253 from skriss/update-license
Update all license headers
2017-12-19 19:26:54 -05:00
Steve Kriss 79c0e5a09a don't error if PVC is not bound to a PV
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-19 14:10:58 -08:00
Steve Kriss 3641c2c043 update all license headers
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-15 13:38:12 -08:00
Steve Kriss 56c9d68137 switch to logrus.FieldLogger and arktest.NewLogger() everywhere
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-13 09:38:00 -08:00
Steve Kriss 4aea9b9a2c switch from Update() to Patch()
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-12 15:23:45 -08:00
Andy Goldstein 992940c55c
Merge pull request #227 from skriss/plugin-logger-fix
Bug fixes: obj/block store plugin logging and remapped namespaces issue
2017-11-30 12:17:56 -05:00
Steve Kriss 121b7153a7 when restoring, don't restore the Namespaces cluster-scoped resource independently
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-30 09:07:38 -08:00
Andy Goldstein bd8f433188
Merge pull request #229 from skriss/plugin-docs
add godoc for ResourceSelector
2017-11-30 11:25:53 -05:00
Steve Kriss 94ac3b3529 godoc for ResourceSelector
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-30 08:20:27 -08:00
Steve Kriss 8a0ac2117b give each plugin its own logrusAdapter
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 14:36:21 -08:00
Steve Kriss cbcd15d603 add/update documentation for plugins
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 14:24:23 -08:00
Steve Kriss b66efd9416 rename NewPluginLogger to NewLogger
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 14:01:31 -08:00
Steve Kriss 038fa39451 remove ListSnapshots and GetAllSnapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 13:59:29 -08:00
Steve Kriss 995050390c add plugin add/remove commands
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 12:22:30 -08:00
Steve Kriss 0b6e78145c
Merge pull request #223 from ncdc/simplify-plugin-serve
Simplify serving plugins for external developers
2017-11-29 11:09:22 -08:00
Steve Kriss 5b8562e73c
Merge pull request #222 from ncdc/fix-log-location-hook
Fix log location hook prefix stripping
2017-11-29 11:07:43 -08:00
Andy Goldstein dc484d1566 Set logger for cloudprovider plugin
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-29 13:54:20 -05:00
Andy Goldstein 1f7e9b65e8 Simply serving plugins for external developers
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-29 13:42:53 -05:00
Andy Goldstein c700455272 Support custom volume snapshots & restores
The main Ark code was hard-coding specific support for AWS, GCE, and
Azure volume snapshots and restores, and anything else was considered
unsupported.

Add GetVolumeID and SetVolumeID to the BlockStore interface, to allow
block store plugins to handle volume snapshots and restores.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-29 13:19:40 -05:00
Andy Goldstein 99f67db39d Fix log location hook prefix stripping
The log location hook was matching github.com/heptio/ark and stripping
off that + 1 more char. This meant that
github.com/heptio/ark-plugin-example/foo.go was being listed as
plugin-example/foo.go instead of
github.com/heptio/ark-plugin-example/foo.go.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-29 12:56:01 -05:00
Andy Goldstein 526b604237 Add boolptr helper package
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-29 12:25:10 -05:00
Andy Goldstein f0b35cc45a
Merge pull request #213 from skriss/restore-action-plugins
Convert restorers to plugins
2017-11-28 14:50:41 -05:00
Steve Kriss 179b95c81d convert restorers to plugins
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-28 10:58:12 -08:00
Andy Goldstein c2dc41efd8 Add backup & schedule describers
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-27 15:22:20 -05:00
Steve Kriss 0f2d1ab82b migrate backup actions to plugins
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-21 10:03:03 -08:00
Steve Kriss fc6da9b3db
Merge pull request #203 from ncdc/log-server-version
Log server version at startup
2017-11-15 12:13:59 -08:00
Nolan Brubaker 194d21c6a3 Update default TTL to 30 days
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2017-11-14 17:03:30 -05:00
Andy Goldstein c57a9b94b9 Log server version at startup
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-14 16:58:14 -05:00
Andy Goldstein b184c0b348 Seek to 0 before uploading restore results
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-14 14:39:42 -05:00
Andy Goldstein 4c481f4d23 Make logSource more concise
Strip off leading .../github.com/heptio/ark/

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-14 14:21:19 -05:00
Andy Goldstein af189fd5f4 Add error & logSource hooks to backup & restore loggers
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-14 14:19:52 -05:00
Andy Goldstein 34a6f492e5 Log restore start/end
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-14 14:19:52 -05:00
Andy Goldstein 38aa43885b Log when starting a backup
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-14 14:19:52 -05:00
Steve Kriss 8ba5a29679 add a logger that can be used within plugins to emit logs into Ark server
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-14 10:38:26 -08:00
Steve Kriss 24ce316788 switch built-in cloud providers to run as plugins
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-14 09:47:36 -08:00
Steve Kriss 3975187d57 add .proto files, generated code, and gen script for object/block stores
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-13 15:43:14 -08:00
Steve Kriss cb49c62aaf implement stream reader for gRPC byte streams
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-13 15:43:13 -08:00
Steve Kriss 21e2019540 rename Block/ObjectStoreAdapter -> Block/ObjectStore
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-08 16:58:47 -08:00
Andy Goldstein 737b6d932a Add more group-resource level logging
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-08 12:08:57 -05:00
Steve Kriss 5e4fc8f84a
Merge pull request #182 from ncdc/include-namespace
Fix missing namespace resource when backing up namespaces with auto cluster resources
2017-11-03 13:56:11 -07:00
Andy Goldstein efa4e57d1e More namespace special casing
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-03 16:34:07 -04:00
Devan Goodwin 9471f9da3c Fix missing namespace metadata after restore.
If backing up specific namespaces with "auto" cluster resources, the
actual namespace objects themselves were not being included in the
backup. Restore would create them but any labels or metadata would be
lost.

Instead handle the special case of namespace as a cluster level resource
we may still need, even if excluding most cluster level resources.

Signed-off-by: Devan Goodwin <dgoodwin@redhat.com>
2017-11-03 12:12:03 -03:00
Andy Goldstein b2d80471ac Move restore warnings/errors to object storage
If you have a large number of warnings and/or errors, the restore
object's size can exceed the maximum allowed by etcd. Move them to
object storage, and add a new describe command to fetch and display them
on the fly.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-02 15:53:59 -04:00
Steve Kriss 55c038afa0 fix restore namespace remapping bug
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-02 10:39:54 -07:00
Andy Goldstein 7f959f0184
Merge pull request #174 from skriss/obj-store-reader
Switch PutObject to take an io.Reader rather than an io.ReadSeeker
2017-11-02 12:57:09 -04:00
Steve Kriss c49d11f17a
Merge pull request #171 from nrb/168-fix
Don't delete 'headless' services
2017-11-01 16:30:32 -07:00
Steve Kriss e7703d88ec switch UploadBackup, UploadRestoreLog, PutObject to take io.Reader
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-01 15:48:13 -07:00
Nolan Brubaker d87e8ee16e Don't delete 'headless' services
Deleting the clusterIP field when the service should be headless will
cause it to be assigned a new IP on restore; instead it should retain
the headless state after restoration.

Fixes #168

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2017-11-01 14:47:08 -07:00
Steve Kriss 4f59b19cdc
Merge pull request #169 from nrb/160-fix
Use label selector in backup, restore, schedule lists
2017-10-31 16:38:07 -07:00
Nolan Brubaker 36a40a0cd3 Use provided options when getting lists
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2017-10-31 16:13:28 -07:00
Steve Kriss 073795715e
Merge pull request #154 from ncdc/backup-download-remove-file-on-error
backup download: remove file if there's an error
2017-10-30 18:49:57 -07:00
Andy Goldstein 0fab2e5e89 Stop passing encoder/decoder to NewHumanReadablePrinter
We only need them if we've got unstructured/unknown data and we want to
convert it to typed objects.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 14:07:58 -04:00
Andy Goldstein c1bc52eb65 Update gofmt
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein aa253bf016 Use generated deep copy functions
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein 293674c40e React to upstream spdy executor changes
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein 322cbc19e6 dynamicResourceClient: fix resourceClient type
dynamicClient.Resource() now returns an interface

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein 15fe87aea3 Fix clientset imports
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein 4957dfce61 Regenerate code
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein 843345f728 Prep types for 1.8 code-generator
Fix genclient tags
Add deepcopy-gen interfaces to top level types

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:30:27 -04:00
Andy Goldstein 43449885a1 React to move to k8s.io/api
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:30:27 -04:00
Akhil Thampy 1ac0303283
Fix #137. Remove clock from volume_snapshot_action.
Signed-off-by: Akhil Thampy <akhilthampy@yahoo.com>
2017-10-26 13:53:52 -05:00
Andy Goldstein 8a53cef7e6 backup download: remove file if there's an error
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-26 11:09:35 -04:00
Andy Goldstein e38b6d0f4c Support cross-compiling for clients
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-24 12:26:26 -04:00
Steve Kriss a7cc58730e add --include-cluster-resources flag to restores (optional, default true)
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-23 10:51:07 -07:00
Steve Kriss e460199536 fix restore log statement
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-23 10:50:55 -07:00
Andy Goldstein 28e7f0851a Add verb-noun command aliases
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-20 13:26:34 -04:00
Andy Goldstein 901f8e1302 Add pod exec backup hooks
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-20 10:20:59 -04:00
Devan Goodwin ed0194c09b Restructure backups for resource prioritization.
Previously the directory structure separated resources depending on
whether or not they were cluster or namespace scoped. All cluster
resources were restored first, then all namespace resources. Priority
did not apply across both and you could not order any namespace
resources before any cluster resources.

This restructure sorts firstly on resource type.

resources/serviceaccounts/namespaces/ns1.json
resources/nodes/cluster/node1.json

This will break old backups as the format is no longer consistent as
announced on the Google group.

Signed-off-by: Devan Goodwin <dgoodwin@redhat.com>
2017-10-17 08:49:05 -03:00
Steve Kriss 4085e473e1 additional deployment/networkpolicy cases for TestBackupResource
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-11 15:05:12 -07:00
Steve Kriss 07b33ace57 add --include-cluster-resources flag to backup create cmd
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-11 15:05:12 -07:00
Steve Kriss 9438a8670a when backing up PVCs, also back up claimed PVs
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-11 09:54:57 -07:00
Steve Kriss fa427ebcfd switch to logrus for per-backup/restore logs
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-10 14:11:21 -07:00
Steve Kriss 7c0d9dcfce add missing licenses
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-10 13:14:03 -07:00
Steve Kriss e7c62b5f38 add file/line numbers to logs
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-10 13:14:03 -07:00
Steve Kriss 9f3ce8ab1a add log-level flag to server command
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-10 13:14:03 -07:00
Steve Kriss 82655b8359 tweak volume snapshot action logging messages
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-10 11:10:50 -07:00
Steve Kriss d27b16370c Merge pull request #102 from ashish-amarnath/support-multi-az-pvs
Preserve AWS & GCP volume AZ when snapshotting and restoring PVs.
2017-10-10 10:08:57 -07:00
ashish-amarnath 6dc81b42f5 Merge branch 'master' into support-multi-az-pvs
Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2017-10-06 23:24:41 -07:00
Ashish Amarnath 9fc9dbb413 Preserve PV's AZ info when snapshotting and restoring PVs.
- Read PV's AZ info from fault-domain label of the PV object for snapshotting.
- Store PV's AZ info in the VolumeInfo.
- Add tests for reading the label from the PV object.
- Remove availability zone validation in AWS and GCP BlockStorageAdaptor.
- Add volumeAZ as a parameter to methods in the BlockStorageAdapter interface.
- Get AZ from VolumeInfo when restoring PV snapshot.
- Remove references to PV availability zone in docs.

Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2017-10-06 23:24:41 -07:00
Justin Nauman 1fd1e8deaa Excluding nodes and endpoints from restoration
- Introduced a blacklist of resources that are non-restorable.  The
goal being that the backup can still include these resources for
logging/auditing purposes but they are explicitly added to
ExcludedResources in the RestorController's "defaulting" logic
to ensure that if someone were to explicitly ask for nodes
that they would be expressly denied.

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-10-05 22:37:29 -05:00
Steve Kriss 9ac71b03cc add logrus hook to log error file/line/function from stack traces
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-05 08:49:33 -07:00
Steve Kriss 914165095a switch logging to logrus and errors to pkg/errors
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-10-05 08:49:33 -07:00
Steve Kriss a5d4d138b2 Merge pull request #96 from ncdc/fix-config-watch
Make config change detection more robust
2017-09-19 09:57:52 -07:00
Andy Goldstein 024f65579c Merge pull request #69 from jrnt30/user-agent
Adding in customized user-agent
2017-09-19 09:51:57 -07:00
Justin Nauman 97f8f2426f Addressing PR feedback and adding tests
Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-09-18 17:42:16 -05:00
Andy Goldstein 84fa48d57a Make config change detection more robust
Fix 2 issues with config change detection:
- Objects received via Get() don't have kind and apiVersion set, while
those from Watch() do, leading to false positives.
- Compare the unmodified config (prior to applying defaults) to the
updated one from Watch().

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-14 17:12:32 -04:00
Andy Goldstein 609bf80f9e Merge pull request #90 from skriss/change-azure-apitimeout
change Azure API timeout default to 2m
2017-09-14 09:38:09 -04:00
Steve Kriss 5424eba417 change Azure API timeout default to 2m
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-13 16:20:12 -07:00
Steve Kriss acb3babb87 when restoring azureDisk from snapshot, update the diskURI with the new diskName
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-13 16:10:30 -07:00
Justin Nauman d76ed7b49e Adding in customized user-agent
Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-09-13 16:35:45 -05:00
Steve Kriss 5405067a2e Merge pull request #88 from ncdc/fix-restore-logs-when-backup-name-has-dash
Support restore logs when backup name has -
2017-09-13 14:14:55 -07:00
Steve Kriss f31af75dc3 Merge pull request #82 from ncdc/tweak-version-output
Tweak version display
2017-09-13 14:14:44 -07:00
Andy Goldstein 136b8f200d Support restore logs when backup name has -
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 16:04:46 -04:00
Andy Goldstein 81fda27440 Tweak version display
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 10:16:16 -04:00
Andy Goldstein b43b65b7bf Revert to prior name format for tarballs
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 09:51:40 -04:00
Andy Goldstein 7b892b63ed make update
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 09:51:40 -04:00
Andy Goldstein e8a88747eb Use pointers when printing schema.GroupResource
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 09:51:40 -04:00
Andy Goldstein 9deb4d4cae backup download: switch to --output as full path
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 09:51:40 -04:00
Andy Goldstein 28fac1b156 Don't unzip backup data when streaming
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 09:51:40 -04:00
Andy Goldstein 273b563c86 Add per-restore logs
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-13 09:51:40 -04:00
Andy Goldstein d0b7880881 Merge pull request #78 from skriss/restore-resource-inc-exc
Add --include-resources, --exclude-resources for restore operations
2017-09-12 15:53:21 -04:00
Steve Kriss 4dfce17de5 add resource includes/excludes for Restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-12 12:43:23 -07:00
Justin Nauman 98d4660d27 Adding in support for backup download
- Adding in support for a new `download` subcommand of backup
- Adjusted signing to allow for multiple types
- Adding in git sha version during build more granular version
  debugging

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-09-11 20:17:33 -05:00
Steve Kriss 907ae6c5b0 move groupresource resolution into discovery helper
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-11 14:41:50 -07:00
Andy Goldstein ab99fcc2e2 Update backup and log file name formats
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:49 -04:00
Andy Goldstein 3bb02ac949 Use ioutil.Discard in unit test
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:49 -04:00
Andy Goldstein 71fadf381e Clarify Backup() param names
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:49 -04:00
Andy Goldstein a583f22ea9 Add ActionContext so backup actions can log
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:48 -04:00
Andy Goldstein 50a95d052e Delete all objects in backup dir
Delete all objects in backup "dir" when deleting a backup, instead of
hard-coding individual file names/types. This way, we'll be able to
delete log files and anything else we add without having to update our
deletion code.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:48 -04:00
Andy Goldstein 193000e487 Add ListObjects (for backup dir deletions)
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:48 -04:00
Andy Goldstein 5c9ffae88f Renerate files
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:48 -04:00
Andy Goldstein 03dde45c09 Add 'ark backup logs' command for retrieval
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:48 -04:00
Andy Goldstein 9848a7a55b Save backup log file to object storage
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:48 -04:00
Andy Goldstein b50a78cf7b Work around glog flag.Parse issue
Due to a change in glog, if you don't call flag.Parse, every log line
prints out 'ERROR: logging before flag.Parse'. This works around that
change. Ultimately we need to switch to a different logging library.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:47 -04:00
Andy Goldstein 860eb7e845 Match Azure sdk changes
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-09-11 15:40:47 -04:00
Steve Kriss 7648155c9a delete restores when GC'ing backups & don't orphan cloud resources
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-06 10:34:08 -07:00
Andy Goldstein 305e7fcea0 Merge pull request #59 from jrnt30/explicit-includes-excludes-restore
Allows explicit include/exclude of namespaces on restores
2017-09-06 11:21:24 -04:00
Andy Goldstein 56c680f991 Merge pull request #62 from jrnt30/default-home-resolution
Allow default kubeconfig resolution
2017-09-06 11:16:10 -04:00
Justin Nauman 2457d8f116 Removing instead of deprecating Namespace attr
- Per discussion, there is no reason to deal
with the complexity of backwards compatibility
with the Namespace attribute on the Restore
domain.

- Also noticed there was an error on the
validation of the BackupController where
the message would actually just be the index
of the error and not the contents of the message
itself.

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-09-05 17:18:10 -05:00
Justin Nauman b50a046370 Additional Validation on Include/Exclude
- Adding in additional test to ensure *Namespaces attributes
don't directly conflict logically with one another
- Additional PR changes around naming/typos

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-09-05 17:17:46 -05:00
Justin Nauman af2a792a9a Allows explicit include/exclude of namespaces on restores
- Introduces similar Include/Exclude declaration on the Restore
resource and cli flags
- Kept support for legacy Namespaces attribute until it could be
deprecating.  Defining both IncludeNamespaces and Namespaces results
in a validation error and the Restore will not be processed (shouldn't
be able to occur)

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-09-05 17:17:41 -05:00
Steve Kriss ba299f880f in GetAllBackups don't error if single backup is unreadable
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-05 09:56:37 -07:00
Andy Goldstein 43c74e33c2 Merge pull request #57 from skriss/restore_missing_backup_fix
during restore, try to get backup directly from obj storage if not in cache/API
2017-09-05 12:47:13 -04:00
Steve Kriss f07a70c604 during restore, try to get backup directly from obj storage if not in cache/API
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-09-05 09:25:39 -07:00
Justin Nauman c0e94a4c1a Allow default kubeconfig resolution
- Changed the default kubeconfig loading to utilize
the client-go's loader strategy.  This allows users
to use the Ark client without having to explicitly
define a KUBECONFIG env var or argument.

This more closely resemebles how Kubectl works and users
are probably more used to while preserving the
current rules.

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-08-28 07:56:57 -05:00
Justin Nauman f017a23d41 AWS Paging Support
- Adding in paging support for the S3 and Snapshot
AWS integration.

As a testing note, you can add in a a MaxKeys to the S3
request as an easy way to ensure that paging is working
properly without having to creation over 1k backups.

Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-08-28 05:49:39 -05:00
Steve Kriss 768aed4ddd make --snapshot-volumes, --restore-volumes optional with sensible default behavior based on PVProvider existence
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-23 09:47:52 -07:00
Steve Kriss 8d5c8ffcbb get rid of StorageAdapter structs and move validation into block/object
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-23 09:46:15 -07:00
Steve Kriss ebc06fd632 make PVProvider optional in server config; disallow snap/restore PVs when not provided
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-23 09:46:15 -07:00
Justin Nauman 3ea9899140 Adding in defaults for backup creation and restore
Signed-off-by: Justin Nauman <justin.r.nauman@gmail.com>
2017-08-21 09:47:11 -05:00
Andy Goldstein c088470ba5 Merge pull request #35 from skriss/server_config_validation
validate cloud-provider config at startup & make PVProvider optional
2017-08-14 16:03:15 -04:00
Steve Kriss 726bbbb393 add zone/location validation to cloudprovider constructors
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-14 12:56:12 -07:00
Andy Goldstein a865cb87ee Merge pull request #37 from skriss/iops_fix
only save/use iops for io1 volumes
2017-08-14 15:24:30 -04:00
Steve Kriss c2271776cc remove unused params from GCP objectStorageAdapter
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-14 10:56:05 -07:00
Steve Kriss aff57e0571 switch to int64 for iops val
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-14 09:42:43 -07:00
Mathias Merscher df320d7bf3
adds support for S3 SSE with KMS
Signed-off-by: Mathias Merscher <Mathias.Merscher@dg-i.net>
2017-08-14 18:18:19 +02:00
Andy Goldstein e966eb9ab0 Merge pull request #38 from skriss/cluster-only-restore-err
check for namespaces/ dir before reading it
2017-08-14 09:46:50 -04:00
Steve Kriss 49809b0a59 check for namespaces/ dir before reading it
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-11 14:05:06 -07:00
Steve Kriss c41c958777 only save/use iops for io1 volumes
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-08-11 13:33:01 -07:00
Andy Goldstein 680e17815b Schedules: treat 1st cron field as minutes
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-08-10 12:44:00 -04:00
Andy Goldstein e0d2a2d4d0 Remove k8s.io/apiextensions-apiserver from vendor
Remove k8s.io/apiextensions-apiserver since we're no longer registering
CRDs in code.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-08-07 12:38:07 -04:00
Andy Goldstein 9dcff62f1a Pull in azure, gcp, oidc client-go auth provider plugins
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-08-07 11:24:22 -04:00
Andy Goldstein 2c9d22771c Use NoError instead of NoErrorf
Prepping to roll back to the same version of testify that client-go
uses, and that does not have NoErrorf.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-08-07 11:20:38 -04:00
Andy Goldstein d5953b3772 Simplify check for owner references
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-08-02 17:10:39 -04:00
Andy Goldstein 2fe501f527 Initial commit
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-08-02 13:27:17 -04:00