122 lines
4.9 KiB
Markdown
122 lines
4.9 KiB
Markdown
# (Experimental) NVIDIA GPU support in minikube
|
||
|
||
minikube has experimental support for using NVIDIA GPUs on Linux.
|
||
|
||
## Using NVIDIA GPUs on minikube on Linux with `--vm-driver=kvm2`
|
||
|
||
When using NVIDIA GPUs with the kvm2 vm-driver. We passthrough spare GPUs on the
|
||
host to the minikube VM. Doing so has a few prerequisites:
|
||
|
||
- You must install the [kvm2 driver](drivers.md#kvm2-driver). If you already had
|
||
this installed make sure that you fetch the latest
|
||
`docker-machine-driver-kvm2` binary that has GPU support.
|
||
|
||
- Your CPU must support IOMMU. Different vendors have different names for this
|
||
technology. Intel calls it Intel VT-d. AMD calls it AMD-Vi. Your motherboard
|
||
must also support IOMMU.
|
||
|
||
- You must enable IOMMU in the kernel: add `intel_iommu=on` or `amd_iommu=on`
|
||
(depending to your CPU vendor) to the kernel command line. Also add `iommu=pt`
|
||
to the kernel command line.
|
||
|
||
- You must have spare GPUs that are not used on the host and can be passthrough
|
||
to the VM. These GPUs must not be controlled by the nvidia/nouveau driver. You
|
||
can ensure this by either not loading the nvidia/nouveau driver on the host at
|
||
all or assigning the spare GPU devices to stub kernel modules like `vfio-pci`
|
||
or `pci-stub` at boot time. You can do that by adding the
|
||
[vendorId:deviceId](https://pci-ids.ucw.cz/read/PC/10de) of your spare GPU to
|
||
the kernel command line. For ex. for Quadro M4000 add `pci-stub.ids=10de:13f1`
|
||
to the kernel command line. Note that you will have to do this for all GPUs
|
||
you want to passthrough to the VM and all other devices that are in the IOMMU
|
||
group of these GPUs.
|
||
|
||
- Once you reboot the system after doing the above, you should be ready to use
|
||
GPUs with kvm2. Run the following command to start minikube:
|
||
```shell
|
||
minikube start --vm-driver kvm2 --gpu
|
||
```
|
||
|
||
This command will check if all the above conditions are satisfied and
|
||
passthrough spare GPUs found on the host to the VM.
|
||
|
||
If this succeeded, run the following commands:
|
||
```shell
|
||
minikube addons enable nvidia-gpu-device-plugin
|
||
minikube addons enable nvidia-driver-installer
|
||
```
|
||
|
||
This will install the NVIDIA driver (that works for GeForce/Quadro cards)
|
||
on the VM.
|
||
|
||
- If everything succeeded, you should be able to see `nvidia.com/gpu` in the
|
||
capacity:
|
||
```shell
|
||
kubectl get nodes -ojson | jq .items[].status.capacity
|
||
```
|
||
|
||
### Where can I learn more about GPU passthrough?
|
||
|
||
See the excellent documentation at
|
||
<https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF>
|
||
|
||
### Why are so many manual steps required to use GPUs with kvm2 on minikube?
|
||
|
||
These steps require elevated privileges which minikube doesn't run with and they
|
||
are disruptive to the host, so we decided to not do them automatically.
|
||
|
||
## Using NVIDIA GPU on minikube on Linux with `--vm-driver=none`
|
||
|
||
NOTE: This approach used to expose GPUs here is different than the approach used
|
||
to expose GPUs with `--vm-driver=kvm2`. Please don't mix these instructions.
|
||
|
||
- Install minikube.
|
||
|
||
- Install the nvidia driver, nvidia-docker and configure docker with nvidia as
|
||
the default runtime. See instructions at
|
||
<https://github.com/NVIDIA/nvidia-docker>
|
||
|
||
- Start minikube:
|
||
```shell
|
||
minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost
|
||
```
|
||
|
||
- Install NVIDIA's device plugin:
|
||
```shell
|
||
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.10/nvidia-device-plugin.yml
|
||
```
|
||
|
||
## Why does minikube not support NVIDIA GPUs on macOS?
|
||
|
||
VM drivers supported by minikube for macOS doesn't support GPU passthrough:
|
||
|
||
- [mist64/xhyve#108](https://github.com/mist64/xhyve/issues/108)
|
||
- [moby/hyperkit#159](https://github.com/moby/hyperkit/issues/159)
|
||
- [VirtualBox docs](http://www.virtualbox.org/manual/ch09.html#pcipassthrough)
|
||
|
||
Also:
|
||
|
||
- For quite a while, all Mac hardware (both laptops and desktops) have come with
|
||
Intel or AMD GPUs (and not with NVIDIA GPUs). Recently, Apple added [support
|
||
for eGPUs](https://support.apple.com/en-us/HT208544), but even then all the
|
||
supported GPUs listed are AMD’s.
|
||
|
||
- nvidia-docker [doesn't support
|
||
macOS](https://github.com/NVIDIA/nvidia-docker/issues/101) either.
|
||
|
||
## Why does minikube not support NVIDIA GPUs on Windows?
|
||
|
||
minikube supports Windows host through Hyper-V or VirtualBox.
|
||
|
||
- VirtualBox doesn't support PCI passthrough for [Windows
|
||
host](http://www.virtualbox.org/manual/ch09.html#pcipassthrough).
|
||
|
||
- Hyper-V supports DDA (discrete device assignment) but [only for Windows Server
|
||
2016](https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment)
|
||
|
||
Since the only possibility of supporting GPUs on minikube on Windows is on a
|
||
server OS where users don't usually run minikube, we haven't invested time in
|
||
trying to support NVIDIA GPUs on minikube on Windows.
|
||
|
||
Also, nvidia-docker [doesn't support
|
||
Windows](https://github.com/NVIDIA/nvidia-docker/issues/197) either.
|