Merge pull request #322 from ncdc/run-server-in-separate-ns

Move ark server & minio to heptio-ark-server ns

README.md

@@ -133,6 +133,18 @@ For more information, see [the debugging information][18].
 
 ### Clean up
 
+Delete any backups you created:
+
+```
+kubectl delete -n heptio-ark backup --all
+```
+
+Before you continue, wait for the following to show no backups:
+
+```
+ark backup get
+```
+
 To remove the Kubernetes objects for this example from your cluster, run:
 
 ```
@@ -147,7 +159,7 @@ kubectl delete -f examples/nginx-app/base.yaml
 
 ## Troubleshooting
 
-If you encounter any problems that the documentation does not address, [file an issue][4] or talk to us on the [Kubernetes Slack team][25] channel `#ark-dr`.
+If you encounter any problems that the documentation does not address, review the [troubleshooting][30] page, [file an issue][4], or talk to us on the [Kubernetes Slack team][25] channel `#ark-dr`.
 
 ## Contributing
 
@@ -197,3 +209,4 @@ See [the list of releases][6] to find out about feature changes.
 [27]: /docs/hooks.md
 [28]: /docs/plugins.md
 [29]: https://heptio.github.io/ark/
+[30]: /docs/troubleshooting.md

docs/aws-config.md

@@ -90,7 +90,7 @@ Create a Secret. In the directory of the credentials file you just created, run:
 
 ```bash
 kubectl create secret generic cloud-credentials \
-    --namespace <ARK_NAMESPACE> \
+    --namespace <ARK_SERVER_NAMESPACE> \
     --from-file cloud=credentials-ark
 ```
 

docs/azure-config.md

@@ -115,7 +115,7 @@ Now you need to create a Secret that contains all the seven environment variable
 
 ```bash
 kubectl create secret generic cloud-credentials \
-    --namespace <ARK_NAMESPACE> \
+    --namespace <ARK_SERVER_NAMESPACE> \
     --from-literal AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} \
     --from-literal AZURE_TENANT_ID=${AZURE_TENANT_ID} \
     --from-literal AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} \

docs/gcp-config.md

@@ -74,7 +74,7 @@ Create a Secret. In the directory of the credentials file you just created, run:
 
 ```bash
 kubectl create secret generic cloud-credentials \
-    --namespace <ARK_NAMESPACE> \
+    --namespace <ARK_SERVER_NAMESPACE> \
     --from-file cloud=credentials-ark
 ```
 

docs/namespace.md

@@ -1,15 +1,30 @@
 # Run in custom namespace
 
-In Ark version 0.7.0 and later, you can run Ark in any namespace. To do so, you specify the namespace in the YAML files that configure the Ark server. You then also specify the namespace when you run Ark client commands.
+In Ark version 0.7.0 and later, you can run Ark in any namespace. To do so, you specify the
+namespace in the YAML files that configure the Ark server. You then also specify the namespace when
+you run Ark client commands.
 
 ## Edit the example files
 
-The Ark repository includes [a set of examples][0] that you can use to set up your Ark server. The examples specify only the default `heptio-ark` namespace. To run in another namespace, you edit the relevant files to specify your custom namespace.
+The Ark repository includes [a set of examples][0] that you can use to set up your Ark server. The
+examples place the server in the `heptio-ark-server` namespace, and backup/schedule/restore/config
+data in the `heptio-ark` namespace.
+
+To run the server in another namespace, you edit the relevant files, changing `heptio-ark-server` to
+your desired namespace.
+
+To store your backups, schedules, restores, and config in another namespace, you edit the relevant
+files, changing `heptio-ark` to your desired namespace.
+
+WARNING: It is recommended to run the Ark server in one namespace, and place your backups, schedules,
+restores, and config in a different namespace. You might encounter issues with deleting a single Ark
+namespace that contains everything.
 
 For all cloud providers, edit `https://github.com/heptio/ark/blob/master/examples/common/00-prereqs.yaml`. This file defines:
 
 * CustomResourceDefinitions for the Ark objects (backups, schedules, restores, configs, downloadrequests)
-* The Ark namespace
+* The namespace where the Ark server runs
+* The namespace where backups, schedules, restores, and the config are stored
 * The Ark service account
 * The RBAC rules to grant permissions to the Ark service account
 
@@ -48,4 +63,4 @@ ark client config set namespace=<NAMESPACE_VALUE>
 
 
 
-[0]: https://github.com/heptio/ark/tree/master/examples
+[0]: https://github.com/heptio/ark/tree/master/examples

docs/troubleshooting.md

@@ -0,0 +1,38 @@
+# Troubleshooting
+
+## heptio-ark namespace stuck terminating / unable to delete backups
+
+Ark v0.7.0 added the ability to delete backups by adding what is called an Ark "finalizer" to each
+backup. When you request the deletion of an object that has at least one finalizer, Kubernetes sets
+the object's "deletion timestamp" (indicating the object has been marked for deletion), but it does
+not immediately delete the object. Instead, Kubernetes only deletes the object when it no longer has
+any finalizers. This means that something (Ark, in this case) must process the backup and then
+remove the Ark finalizer from it.
+
+Ark versions before v0.7.1 place the Ark server pod in the same namespace as backups, restores,
+schedules, and the Ark config. If you try to delete the namespace (`kubectl delete
+namespace/heptio-ark`), it's possible that the Ark server pod is deleted before the backups, because
+the order of deletions is arbitrary. If this happens, the remaining bacukps will be "stuck"
+deleting, because the Ark server pod no longer exists to remove their finalizers.
+
+With v0.7.1, we strongly encourage you to run the Ark server pod in a different namespace than the
+one used for backups, schedules, restores, and the Ark config. This is the default configuration as
+of v0.7.1.
+
+If you encounter this problem, here is how to fix it. First, make sure you have `jq` installed. Then
+run:
+
+```
+bash <(kubectl -n heptio-ark get backup -o json | jq -c -r $'.items[] | "kubectl -n heptio-ark patch backup/" + .metadata.name + " -p \'" + (({metadata: {finalizers: ( (.metadata.finalizers // []) - ["gc.ark.heptio.com"]), resourceVersion: .metadata.resourceVersion}}) | tostring) + "\' --type=merge"')
+```
+
+This retrieves a list of backups and uses it to generate and run a list of commands that look like:
+
+```
+kubectl -n heptio-ark patch backup/my-backup -p '{"metadata":{"finalizers":[],"resourceVersion":"461343"}}' --type=merge
+kubectl -n heptio-ark patch backup/some-other-backup -p '{"metadata":{"finalizers":[],"resourceVersion":"461718"}}' --type=merge
+```
+
+If you receive errors that patching backups is not allowed, it's possible that the Ark
+CustomResourceDefinitions (CRDs) were deleted. You'll need to recreate them (they're in
+`examples/common/00-prereqs.yaml`), then follow the steps above.

examples/azure/00-ark-deployment.yaml

@@ -16,7 +16,7 @@
 apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   name: ark
 spec:
   replicas: 1
@@ -32,6 +32,8 @@ spec:
           image: gcr.io/heptio-images/ark:latest
           command:
             - /ark
+            - --namespace
+            - heptio-ark
           args:
             - server
           envFrom:

examples/common/00-prereqs.yaml

@@ -93,12 +93,18 @@ kind: Namespace
 metadata:
   name: heptio-ark
 
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: heptio-ark-server
+
 ---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: ark
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   labels:
     component: ark
 
@@ -111,42 +117,9 @@ metadata:
     component: ark
 subjects:
   - kind: ServiceAccount
-    namespace: heptio-ark
+    namespace: heptio-ark-server
     name: ark
 roleRef:
   kind: ClusterRole
   name: cluster-admin
   apiGroup: rbac.authorization.k8s.io
-
----
-apiVersion: rbac.authorization.k8s.io/v1beta1
-kind: Role
-metadata:
-  namespace: heptio-ark
-  name: ark
-  labels:
-    component: ark
-rules:
-  - apiGroups:
-      - ark.heptio.com
-    verbs:
-      - "*"
-    resources:
-      - "*"
-
----
-apiVersion: rbac.authorization.k8s.io/v1beta1
-kind: RoleBinding
-metadata:
-  namespace: heptio-ark
-  name: ark
-  labels:
-    component: ark
-subjects:
-  - kind: ServiceAccount
-    namespace: heptio-ark
-    name: ark
-roleRef:
-  kind: Role
-  name: ark
-  apiGroup: rbac.authorization.k8s.io

examples/common/10-deployment.yaml

@@ -16,7 +16,7 @@
 apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   name: ark
 spec:
   replicas: 1
@@ -34,6 +34,8 @@ spec:
             - /ark
           args:
             - server
+            - --namespace
+            - heptio-ark
           volumeMounts:
             - name: cloud-credentials
               mountPath: /credentials

examples/minio/00-minio-deployment.yaml

@@ -16,7 +16,7 @@
 apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   name: minio
   labels:
     component: minio
@@ -54,7 +54,7 @@ spec:
 apiVersion: v1
 kind: Service
 metadata:
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   name: minio
   labels:
     component: minio
@@ -71,7 +71,7 @@ spec:
 apiVersion: v1
 kind: Secret
 metadata:
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   name: cloud-credentials
   labels:
     component: minio
@@ -85,7 +85,7 @@ stringData:
 apiVersion: batch/v1
 kind: Job
 metadata:
-  namespace: heptio-ark
+  namespace: heptio-ark-server
   name: minio-setup
   labels:
     component: minio

examples/minio/10-ark-config.yaml

@@ -24,7 +24,7 @@ backupStorageProvider:
   config:
     region: minio
     s3ForcePathStyle: "true"
-    s3Url: http://minio.heptio-ark.svc:9000
+    s3Url: http://minio.heptio-ark-server.svc:9000
 backupSyncPeriod: 1m
 gcSyncPeriod: 1m
 scheduleSyncPeriod: 1m