* Correcting Openshift on IBM Documentation Error
I have to admit to some significant error and embarrassment regarding the documentation update
about Openshift on IBM Cloud pull request https://github.com/vmware-tanzu/velero/pull/8069.
I will correct my error before it gets any further.
Just exposing /var/data/kubelet/pods is incorrect and host path /var/lib/kubelet/pods should remain unchanged.
The errors with the defaults during csi snapshot data movement were:
data path backup failed: Failed to run kopia backup: unable to get local
block device entry: resolveSymlink: lstat /var/data/: no such
file or directory
I suspected this was the same as RancherOS and Nutanix. It is not.
The original tested changes changed both /var/lib/kubelet/{pods,plugins} to
/var/data/kubelet/{pods,plugins}.
The published changes only result in the error:
```
status:
completionTimestamp: '2024-08-02T17:12:29Z'
message: >-
data path backup failed: Failed to run kopia backup: unable to get local
block device entry: resolveSymlink: lstat /var/data/kubelet/plugins: no such
file or directory
node: 10.240.0.5
phase: Failed
progress: {}
startTimestamp: '2024-08-02T17:12:11Z'
```
After making continued modifications to the daemonset the correct configuration was:
```
volumeMounts:
- name: host-pods
mountPath: /host_pods
mountPropagation: HostToContainer
- name: host-plugins
mountPath: /var/data/kubelet/plugins
mountPropagation: HostToContainer
```
```
volumes:
- name: host-pods
hostPath:
path: /var/lib/kubelet/pods
type: ''
- name: host-plugins
hostPath:
path: /var/data/kubelet/plugins
type: ''
```
Only the changes to the plugin path were required.
The plugin path changes were required to both the mount path and the host path.
Regardless of whether /var/lib/kubelet/pods or /var/data/kubelet/pods host path, backups and restore
succeeded provided the plugin path was modified.
```
volumeMounts:
- name: host-pods
mountPath: /host_pods
mountPropagation: HostToContainer
- name: host-plugins
mountPath: /var/data/kubelet/plugins
mountPropagation: HostToContainer
```
```
volumes:
- name: host-pods
hostPath:
path: /var/data/kubelet/pods
type: ''
- name: host-plugins
hostPath:
path: /var/data/kubelet/plugins
type: ''
```
After getting on-host access was able to confirm. Pods are at /var/lib/kubelet/pods.
```
ls /var/lib/kubelet/pods
07c0be63-335d-4cfb-b39f-816bc2fb32cd
51f31b3e-4710-4ef0-8626-5f1a78a624b2
a4802fd3-3b62-45a4-8f21-974880b6f92a
cccb35c9-b4f9-4ca9-a697-736ae64f09ad
0a5d4366-7fa1-4525-9e45-a43a362b8542
558b0643-0661-4d4a-b03e-aac60c6ad710
a4b106fb-5b7b-48e5-828a-ea7b41ba0e59
ce1290e1-4330-4df6-8166-14784bcce930
```
On host the volumes are in /var/data/kubelet/plugins.
```
ls /var/data/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/231e04896c4f528efb95d23a3c153db9fc4a7206b7320f74443f30de7228dba5/globalmount/velero/backups/backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c/
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotclasses.json.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-resource-list.json.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotcontents.json.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-results.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshots.json.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-volumesnapshots.json.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-itemoperations.json.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c.tar.gz
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-logs.gz
velero-backup.json
backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-podvolumebackups.json.gz
```
With the volume config changed to expose /var/data/kubelet/plugins for the plugin hostPath, the DataUploads and DataDownloads
succeed for both Filesystem and Block mode PVCs.
```
status:
completionTimestamp: '2024-08-02T17:23:33Z'
node: 10.240.0.5
path: >-
/host_pods/7fcb9d56-7885-437c-acd3-67db6b1ee8ae/volumeDevices/kubernetes.io~csi/pvc-47b91f56-db8c-44bf-9ecc-737170561b4b
phase: Completed
progress:
bytesDone: 5368709120
totalBytes: 5368709120
snapshotID: 8faae36b3592fee4efbfad024f26033e
startTimestamp: '2024-08-02T17:21:22Z'
```
```
status:
completionTimestamp: '2024-08-02T18:42:19Z'
node: 10.240.0.5
phase: Completed
progress:
bytesDone: 5368709120
totalBytes: 5368709120
startTimestamp: '2024-08-02T18:41:00Z'
```
My apologies for the error.
Signed-off-by: Michael Fruchtman <msfrucht@us.ibm.com>
* Add context to plugins mountPath
Signed-off-by: Michael Fruchtman <msfrucht@us.ibm.com>
---------
Signed-off-by: Michael Fruchtman <msfrucht@us.ibm.com>
Updates the documentation for CSI snapshot data movement for OpenShift
on IBM Cloud.
The default hostpath /var/lib/kubelet/pods cannot find
PersistentVolumeClaims with volumeMode: Block on host.
The correct hostpath for OpenShift on IBM Cloud is
/var/data/kubelet/pods.
Signed-off-by: Michael Fruchtman <msfrucht@us.ibm.com>
Lyndon-Li
Shubham Pampattiwar
Michael Steven Fruchtman
Michael Fruchtman
Xun Jiang
qiuming
Xun Jiang/Bruce Jiang
Andy Arnold
Yang Gang
Raghuram Devarakonda
David Zaninovic
lyndon