Commit Graph

303 Commits (51298f84cccaa28d78f013e8096d863f2e820d7f)

Author SHA1 Message Date
Andy Goldstein 2fde1f5fc1
Merge pull request #500 from nrb/fix-497
Exit when failing to update a backup's phase
2018-05-17 14:23:41 -04:00
Andy Goldstein 3a746a3f73
Merge pull request #496 from nrb/fix-477
Check restore Phase before downloading logs
2018-05-17 13:23:23 -04:00
Andy Goldstein f288902e3e
Merge pull request #490 from skriss/jsonpatch
Use JSON merge patches everywhere
2018-05-17 13:02:47 -04:00
Nolan Brubaker 7eac6675e8 Exit when failing to update a backup's phase
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-17 12:47:38 -04:00
Nolan Brubaker f6761ddd00 Check restore Phase before downloading logs
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-17 12:20:55 -04:00
Steve Kriss 5d74a92cf1
Merge pull request #484 from nrb/move-groupresources
Consolidate commonly used GroupResource objects
2018-05-15 14:02:29 -07:00
Nolan Brubaker f936c55a37 Consolidate commonly used GroupResource objects
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-15 15:55:15 -04:00
Andy Goldstein ad93135adb
Merge pull request #483 from skriss/graceful-shutdown-on-sigs
shutdown gracefully on SIGINT/SIGTERM
2018-05-15 14:33:09 -04:00
Andy Goldstein 67263d2652
Merge pull request #486 from nrb/fix-475
Filter on resources that support get & delete
2018-05-15 10:58:56 -04:00
Nolan Brubaker 4fcd222777 Expand required resource verbs with get and delete
Some resources use GET for listing, which resulted in errors.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-15 10:55:03 -04:00
Steve Kriss 6d6f734bc9 use json merge patches
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-14 14:34:24 -07:00
Steve Kriss 1c950aa17b unit test to ensure new cohabitatingResources map for each backup
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-14 10:23:21 -07:00
Steve Kriss ed7fbc9178 use new cohabitatingResources map for each backup
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-11 16:30:14 -07:00
Steve Kriss 20f56e9868 cleanup plugin subprocesses on server exit
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-11 09:48:46 -07:00
Steve Kriss 09c20b51e6 shutdown gracefully on SIGINT/SIGTERM
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-10 16:22:14 -07:00
Steve Kriss fbb5ead4e9 add replicasets and daemonsets to cohabitating resources
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-10 14:15:01 -07:00
Andy Goldstein 43b1f9a19e
Merge pull request #418 from skriss/refactor-patch-tests
use typed structs for decoding patch JSON in unit tests
2018-05-09 15:44:21 -04:00
Andy Goldstein ea83ed32f5
Merge pull request #470 from skriss/backup-cluster-roles-and-bindings
backups: include clusterroles/bindings that reference serviceaccounts
2018-05-09 13:21:13 -04:00
Steve Kriss 041cfc2173 backups: include clusterroles/bindings that reference serviceaccounts
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-09 10:03:55 -07:00
Andy Goldstein ef5ac7fd05
Merge pull request #463 from nrb/fix-454
Don't restore completed pods or jobs
2018-05-09 09:59:34 -04:00
Steve Kriss e81de2491f remove duplicate github.com/satori/go.uuid dependency
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 10:02:59 -07:00
Steve Kriss 51928e9177 use typed structs for decoding patch JSON in unit tests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 09:22:20 -07:00
Steve Kriss 6754955bcd when syncing backups, set their namespace to current cluster's Ark ns
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 09:12:51 -07:00
Nolan Brubaker 923870390b Skip completed jobs and pods when restoring
Completed jobs and pods may be useful in the backup for auditing
purposes, but don't recreate them when restoring.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-02 12:16:54 -04:00
Andy Goldstein 18d6b233da
Merge pull request #453 from nrb/fix-444
Make empty excludes string more accurate
2018-04-24 14:53:33 -04:00
Andy Goldstein c8989231eb
Merge pull request #417 from skriss/upgrade-k8s-deps
Upgrade apimachinery, client-go, api, kubernetes dependencies
2018-04-24 14:53:11 -04:00
Nolan Brubaker 05cb059b1a Make empty excludes string more accurate
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-24 14:34:56 -04:00
Steve Kriss 72d7e5e01d azure: pass full snapshot name when restoring disk
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-23 12:51:05 -07:00
Steve Kriss a466208975 add backwards-compatibility for azure snapshot ID
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-23 11:34:37 -07:00
Steve Kriss 3dc093c24a regenerate code with updated generator
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:59:22 -07:00
Steve Kriss 989169dcfe remove unused kubernetes/pkg/util/version dependency
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:59:22 -07:00
Steve Kriss 97e52f2b3c add timeout arg to rest.NewRequest()
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:58:53 -07:00
Steve Kriss 961d7f2924 kubernetes/pkg/printers/ShortHumanDuration -> apimachinery/pkg/util/duration
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-20 12:58:53 -07:00
Nolan Brubaker 300a010959 Remove finalizers from backups before syncing
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-18 17:32:39 -04:00
Andy Goldstein 730ddb2113
Merge pull request #391 from containscafeine/handle-restore-missing-error
Introduce validation logic to `ark restore logs`
2018-04-12 11:07:29 -04:00
Shubham Minglani 7f3e88151b Add bash and zsh completion support
This commit adds support for auto completion for bash and zsh
shells. A new root level command called "completion" has been
introduced, and the user can get the auto completion code by
running `ark completion bash/zsh`.

For bash completion, the built-in GenBashCompletion() from cobra
has been used, but for zsh, the built-in GenZshCompletion() is
known to cause issues. The workaround has been copied from zsh
completion code of kubectl.

Signed-off-by: Shubham <shubham@linux.com>
2018-04-12 13:26:23 +05:30
Shubham 468c4faf1b Introduce validation logic to `ark restore logs`
This commit introduces validation logic to `ark restore logs`
command, the way it already exists in other commands like `ark
restore create`.

Before the logs for a restore are fetched from the server, the
server is contacted to check if the specified restore exists. If
it does not, it errors out.

Fixes #389

Signed-off-by: Shubham <shubham@linux.com>
2018-04-12 13:06:32 +05:30
Andy Goldstein 0998f6d8af
Merge pull request #405 from nrb/ignore-duplicates
Compare for duplicates before logging object exists errors
2018-04-11 09:47:37 -04:00
Nolan Brubaker 33cc85cd0c Compare backup and cluster objects before logging
When restoring resources that raise an already exists error, check their
equality before logging a message on the restore. If they're the same
except for some metadata, don't generate a message.

The restore process was modified so that if an object had an empty
namespace string, no namespace key is created on the object. This was to
avoid manipulating the copy of the current cluster's object by adding
the target namespace.

There are some cases right now that are known to not be equal via this
method:

- The `default` ServiceAccount in a namespace will not match, primarily
because of differing default tokens. These will be handled in their own
patch
- IP addresses for Services are recorded in the backup object, but are
either not present on the cluster object, or different. An issue for
this already exists at https://github.com/heptio/ark/issues/354
- Endpoints have differing values for `renewTime`. This may be
insubstantial, but isn't currently handled by the resetMetadataAndStatus
function.
- PersistentVolume objects do not match on spec fields, such as
claimRef and cloud provider persistent disk info

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-10 14:55:48 -04:00
Andy Goldstein 258f3e011e
Merge pull request #409 from skriss/fix-398
get backup name from restore spec when handling restore DownloadRequests
2018-04-10 13:29:07 -04:00
Steve Kriss 0d2b49acce get backup name from restore spec when handling restore DownloadRequests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 10:50:10 -07:00
Steve Kriss 5bb47d2b7a extract a helper for comparing slices of Actions reliably
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 09:39:33 -07:00
Andy Goldstein 3f2e222ae4 Don't allow deletion of in-progress backups
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-06 13:08:39 -04:00
Andy Goldstein 644a75e3c1 Set DeleteBackupRequest labels if missing
When the BackupDeletionController processes a request, set the request's
backup-name and backup-uid labels if they aren't currently set.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 15:38:44 -04:00
Andy Goldstein ef57a44827 Require DeleteBackupRequest.Spec.BackupName
Make sure a DeleteBackupRequest has its Spec.BackupName filled in. If
not, record an error in the status and mark the request as processed.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 14:53:10 -04:00
Andy Goldstein 4328b67f93 backup delete/describe enhancements
Always request DeleteBackupRequests for a given backup so we can show
failed deletion attempts if you try to delete a backup that has PV
snapshots when Ark doesn't have a persistentVolumeProvider configured.

When creating a DeleteBackupRequest, include a label for the UID so we
can match based on name and UID when associated DeleteBackupRequests
with a given backup.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 14:21:45 -04:00
Andy Goldstein 1ae492da6c Expire processed deleted backup requests > 24hr
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 11:19:11 -04:00
Andy Goldstein 74f60b1ee1 Switch backup finalizer to DeleteBackupRequest
We ran into a lot of problems using a finalizer on the backup to allow
the Ark server to clean up all associated backup data when deleting a
backup.

Users also found it less than desirable that deleting the heptio-ark
namespace resulted in all the backup data being deleted.

This removes the finalizer and replaces it with an explicit
DeleteBackupRequest that is created as a means of requesting the
deletion of a backup and all its associated data. This is what `ark
backup delete` does.

If you use kubectl to delete a backup or to delete the heptio-ark
namespace, this no longer deletes associated backups. Additionally, as
long as the heptio-ark namespace still exists, the Ark server's
BackupSyncController will continually sync backups into the heptio-ark
namespace from object storage.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 11:16:15 -04:00
Steve Kriss 80b66434c0 move getting client into Complete()
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-03 20:43:42 -07:00
Steve Kriss c60e47dedd use cobra's arg-count validation & call Complete() before Validate()
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-03 09:45:29 -07:00
Shubham 73499c2cff Add limitranges to defaultResourcePriorities
This commit adds limitranges to defaultResourcePriorities as
suggested in #385.

This is done so that pods are not restored before the LimitRange
objects, because that would lead to pods not honoring the requests
and limits set in LimitRange objects.

Fixes #385

Signed-off-by: Shubham <shubham@linux.com>
2018-03-17 21:13:44 +05:30
Shubham 560f9504e1 Fix typo: replace ; with :
This commit replaces a ; with a : in an error message

Signed-off-by: Shubham <shubham@linux.com>
2018-03-16 18:24:34 +05:30
Andy Goldstein 3172e907a1
Merge pull request #341 from skriss/snapshot-tags
Add tags to snapshots, restored volumes
2018-03-14 12:50:40 -04:00
Andy Goldstein 07fcc927b7
Merge pull request #359 from skriss/reduce-azure-polling-interval
Azure: reduce client polling interval from default 60s to 5s
2018-03-14 12:17:54 -04:00
Steve Kriss 45cee7d222 Azure: reduce client polling interval from default 60s to 5s
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-14 09:07:20 -07:00
Steve Kriss b2b16b3c85 Azure: fix bug preventing snapshot deletion
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 17:03:14 -07:00
Steve Kriss 0388845bca AWS: tag snapshots during create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss a05ae1a7cf add useful Ark tags to snapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss df985bca74 Azure: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss cab904570f GCP: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 11:52:09 -07:00
Steve Kriss 9673e9d158 AWS: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 11:52:09 -07:00
Steve Kriss 51c546786e
Merge pull request #374 from sanketjpatel/feat/exclude-events-restore
Add events to nonRestorableResources and cohabitatingResources
2018-03-13 11:23:17 -07:00
Andy Goldstein f83b1de1dc
Merge pull request #356 from skriss/store-azure-snapshot-uri
Azure: store snapshot URI to support cross-resource group restores
2018-03-13 13:45:48 -04:00
Steve Kriss 409f17361d Azure: store snapshot URI to support cross-resource group restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 09:18:20 -07:00
Sanket Patel 7257a75f82 Add events to nonRestorableResources and cohabitatingResources
Fixes #367
Fixes #368

Signed-off-by: Sanket Patel <sanketpatel.301090@gmail.com>
2018-03-12 19:55:32 -05:00
Andy Goldstein 178a007b70
Merge pull request #362 from skriss/tweak-hook-logging
during backup, only log if hooks are actually executing
2018-03-07 13:37:03 -05:00
Steve Kriss 26944cf9a2 only log if hooks are actually executing
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-07 09:29:27 -08:00
Andy Goldstein 7328fd0e10
Merge pull request #343 from skriss/gc-bug
add an AddFunc to handle finalizers in GC for initial lists/resyncs
2018-03-06 16:41:57 -05:00
Andy Goldstein 331e0c28cc Remove Azure location requirement
Instead of requiring the Ark admin to specify a "location" in the azure
persistentVolumeProvider config (meaning only a single location is
supported), get info about the disk (for its location) when creating a
snapshot, and get info about the snapshot (for its location) when
creating a disk from a snapshot.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-03-05 15:20:58 -05:00
Steve Kriss c281124cba add --from-backup flag to ark restore create & allow restore name
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-05 10:06:43 -08:00
Andy Goldstein 6930c846e5 Fix incorrect return when err != nil
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-03-05 10:57:23 -05:00
Andy Goldstein cc9be4489a
Merge pull request #345 from skriss/gcp-project
extract GCP project ID from creds file, remove from config
2018-03-05 10:35:39 -05:00
Steve Kriss cbff5ef53b extract GCP project ID from creds file, remove from config
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-02 09:15:12 -08:00
Steve Kriss 509431909a add an AddFunc to handle finalizers in GC for initial lists/resyncs
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-02-28 16:56:23 -08:00
Steve Kriss 975dce929c
Merge pull request #331 from lypht/schedule
Include cron examples for schedule creation
2018-02-28 15:32:28 -08:00
Josh Hull f733869142
Include cron examples for schedule creation
Signed-off-by: Josh Hull <josh.hull@lypht.com>
2018-02-28 16:50:52 -06:00
Andy Goldstein e618e0e456 server: allow configurable plugin dir
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-27 14:50:38 -05:00
Andy Goldstein f13b0c00a3 server: don't assume /ark for internal plugins
If you want to test changes to the ark server without having to rebuild
and redeploy the ark container, this change allows you to do something
like this (assuming you've created your cloud credentials file):

AWS_SHARED_CREDENTIALS_FILE=credentials-minio ark server -n heptio-ark

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-27 14:32:45 -05:00
Nolan Brubaker e73ba839f3
Merge pull request #296 from blakebarnett/bdb/add_kubecontext_flag
Adds a kubecontext global flag
2018-02-26 17:05:59 -05:00
Nolan Brubaker abeeee6643
Merge pull request #333 from ncdc/add-force-delete-backup
Allow forced backup deletion
2018-02-26 16:44:37 -05:00
Andy Goldstein d24fb232cc Allow forced backup deletion
Add --force and --confirm to `ark backup delete` to support forced
backup deletion. This forcibly removes the Ark GC finalizer (if it's
present) from a backup and will orphan any resources associated with the
backup, such as backup tarballs in object storage, persistent volume
snapshots, and restores for the backup.

If a backup has a deletion timestamp, display `Deleting` in `ark backup
describe` and `ark backup get`.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-26 16:25:22 -05:00
Blake 465c60b408
Add global kubecontext CLI flag
Signed-off-by: Blake <blake.barnett@postmates.com>
2018-02-26 10:47:39 -08:00
Andy Goldstein cc4d2916fa Error if backup missing when creating a restore
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-05 13:29:48 -05:00
Andy Goldstein b2cd8e1fe8 Create the config directory in case it's missing
Always try to create the config directory when saving the client config
in case it doesn't exist.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-02 16:01:52 -05:00
Andy Goldstein 6712e2a9ec
Merge pull request #270 from nrb/fix-232
Clarify backup and restore creation messages
2018-01-11 12:22:11 -05:00
Andy Goldstein f5f1dbac5d
Merge pull request #271 from nrb/fix-246
Invalidate restores based on non-existent backups
2018-01-10 13:24:53 -05:00
Steve Kriss af81f589f6 update GCP block store to use v1 library, and tag snapshots on create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-08 09:23:32 -08:00
Nolan Brubaker 37f498cd37 Clarify backup and restore creation messages
When running `ark <resource> create`, a request is sent to the server,
but the status is not immediately known. Inform the user that a request
was sent and provide a way to get more information on it.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-01-08 10:26:51 -05:00
Nolan Brubaker bb7b0a3c33 Invalidate restores based on missing backups
When creating a restore based on a backup that doesn't exist, the
restore should be marked as invalid and the error clearly communicated
so the user understands why the restore wasn't made.

Previously, the restore was left as in progress with an error attached.

Since restores are CRDs and must be updated via a controller, there's
currently not a way to give the client immediate errors.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-01-08 10:10:01 -05:00
Steve Kriss b53ee44176 update GCP object store to use new/recommended lib
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-05 14:21:13 -08:00
Steve Kriss e130011d1e
Merge pull request #272 from ncdc/arbitrary-namespaces
Support running in any namespace
2018-01-05 14:15:18 -08:00
Andy Goldstein 816f14c0b4 Support running in any namespace
Add the ability for the Ark server to run in any namespace.

Add `ark client config get/set` for manipulating the new client
configuration file in $HOME/.config/ark/config.json. This holds client
defaults, such as the Ark server's namespace (to avoid having to specify
the --namespace flag all the time).

Add a --namespace flag to all client commands.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-05 16:51:59 -05:00
Andy Goldstein de785af89d Support pre and post hooks.
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-05 10:47:34 -05:00
Andy Goldstein d56b5f2b66 Updated generated CRD code
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-04 10:44:20 -05:00
Steve Kriss 656428d0b9
Merge pull request #250 from ncdc/backup-controller-do-as-much-as-possible
BackupController: do as much as possible
2018-01-03 12:10:51 -08:00
Andy Goldstein 1b124a3146 Log backup item action error when it occurs
If a backup item action errors, log the error as soon as it occurs, so
it's clear when the error happened. Also include information about the
groupResource, namespace, and name of the item in the error.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-01-03 13:20:05 -05:00
Andy Goldstein 6b0b6373da
Merge pull request #252 from skriss/delete-backup
add delete backup cmd using finalizer and simplify GC process
2018-01-02 14:26:03 -05:00
Andy Goldstein 56ca4bab2f
Merge pull request #268 from skriss/2018
update copyright header to 2018 and make language/casing consistent
2018-01-02 14:16:26 -05:00
Steve Kriss 1503796419 make language and casing consistent across copyrights
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-02 10:51:49 -08:00
Steve Kriss 3b157afbbd update copyright year to 2018 for generated files
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-02 10:48:42 -08:00