* krunkit: Add krunkit driver
krunkit is a tool to launch configurable virtual machines using the
libkrun platform, optimized for GPU accelerated virtual machines and AI
workloads on Apple silicon.
It is mostly compatible with vfkit; the driver is a simplified copy of
the vfkit driver. Unlike vfkit, krunkit is available only on Apple
silicon.
Changes compared to vfkit driver:
- krunkit requires unix socket for networking, so we must use
vment-helper.
- krunkit does not support HardStop, so we kill it using SIGKILL.
- We must enable vmnet offloading, required for krunkit.
- The code was simplified since vmnet-helper is always used
- Code was cleaned up to use .ResolveStorePath()
- Unused Upgrade() function was removed
- Types and functions that should not be public made private
We require krunkit 0.2.2, supporting --restul-uri=unix://.
* reason: Make vment-helper error driver agnostic
Previously it was used only for vfkit, so we suggested to fallback to
the `nat` network. This advice is not relevant to krunkit or to qemu
(which can also use vmnet-helper).
Change the error to recommend installing vment-helper. We need to think
how we can recommend other networks for vfkit and qemu. Another solution
is to create error for every driver+network combination but this seems
hard to manage.
* hack: Add krunkit integration test
This is the same way that we test vfkit. This test is not running in the
CI.
Issues:
- Need to install and configure vment-helper (requires root).
* site: Add krunkit driver documentation
* vfkit: Log serial console to file
To make debugging easier, add virtio-serial device logging serial
console to file:
~/.minikube/machines/NAME/serial.log
To enable logging, we need to enable the console in the kernel command
line, since we still use direct kernel boot.
Example log:
% cat /Users/nir/.minikube/machines/vfkit/vfkig.log
[ 0.896094] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 0.897186] loop: module loaded
[ 0.897670] virtio_blk virtio2: [vda] 840488 512-byte logical blocks (430 MB/410 MiB)
[ 0.897733] vda: detected capacity change from 0 to 430329856
[ 0.898460] virtio_blk virtio3: [vdb] 40960000 512-byte logical blocks (21.0 GB/19.5 GiB)
[ 0.898533] vdb: detected capacity change from 0 to 20971520000
...
[ 1.794714] systemd[1]: Detected virtualization vm-other.
[ 1.794752] systemd[1]: Detected architecture arm64.
Welcome to Buildroot 2025.02!
[ 1.794944] systemd[1]: Hostname set to <minikube>.
[ 1.795011] systemd[1]: Initializing machine ID from random generator.
...
[ OK ] Started Container Runtime Interface for OCI (CRI-O).
[ OK ] Reached target Multi-User System.
Welcome to minikube
vfkit login: [ 6.681578] systemd-ssh-generator[630]: Binding SSH to AF_UNIX socket /run/ssh-unix-local/socket.
* vfkit: Use EFI bootloader
With the fixed iso, we can simplify the driver using the EFI bootloader
option[1] instead of the legacy and deprecated --kernel, --kernel-cmdline,
and --initrd options[2].
Example run:
% minikube start -p vfkit --driver vfkit --container-runtime containerd --network vmnet-shared
😄 [vfkit] minikube v1.36.0 on Darwin 15.5 (arm64)
✨ Using the vfkit driver based on user configuration
👍 Starting "vfkit" primary control-plane node in "vfkit" cluster
🔥 Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
📦 Preparing Kubernetes v1.33.1 on containerd 1.7.23 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "vfkit" cluster and "default" namespace by default
Comparing direct kernel boot and --bootloader efi shows that it is little bit faster and boot time is more consistent.
% hyperfine -r 10 -C "minikube delete" \
"vfkit-efi/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes" \
"vfkit-direct/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes"
Benchmark 1: vfkit-efi/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes
Time (mean ± σ): 10.205 s ± 0.656 s [User: 0.381 s, System: 0.266 s]
Range (min … max): 9.106 s … 11.254 s 10 runs
Benchmark 2: vfkit-direct/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes
Time (mean ± σ): 10.933 s ± 1.616 s [User: 0.402 s, System: 0.406 s]
Range (min … max): 9.155 s … 14.168 s 10 runs
Summary
vfkit-efi/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes ran
1.07 ± 0.17 times faster than vfkit-direct/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes
[1] https://github.com/crc-org/vfkit/blob/main/doc/usage.md#efi-bootloader
[2] https://github.com/crc-org/vfkit/blob/main/doc/usage.md#deprecated-options
* docs: Update vfkit driver documentation
- Separate vfkit requirements and vmnet-shared requirements
- Update minimal macOS version required for --bootloader efi
- Simplify vfkit upgrade, it is available in brew now
* vfkit: Fix help text for --extra-disks
vfkit can use extra disks but we forgot to mention it in the help text.
* vfkit: Remove stale tap
vfkit in cfergeau/crc will no longer be updated now that it’s in the main repo.
* vfkit: Fix status in index
The vfkit driver is preferred now.
* vfkit: Remove temporary variable
Remove temporary and unneeded mac variable. It is easier to follow the
code when we use d.MACAddress.
* vfkit: Promote state change to INFO level
System state changes should be more visible to make debugging easier.
* vmnet: Add vmnet package
The package manages the vmnet-helper[1] child process, providing
connection to the vmnet network without running the guest as root.
We will use vmnet-helper for the vfkit driver, which does not have a way
to use shared network, when guests can access other guest in the
network. We can use it later with the qemu driver as alternative to
socket_vmnet.
[1] https://github.com/nirs/vmnet-helper
* vfkit: add vmnet-shared network
Add new network option for vfkit "vmnet-shared", connecting vfkit to the
vmnet shared network. Clusters using this network can access other
clusters in the same network, similar to socket_vmnet with QEMU driver.
If network is not specified, we default to the "nat" network, keeping
the previous behavior. If network is "vmnet-shared", the vfkit driver
manages 2 processes: vfkit and vmnet-helper.
Like vfkit, vmnet-helper is started in the background, in a new process
group, so it not terminated if the minikube process group is terminate.
Since vmnet-helper requires root to start the vmnet interface, we start
it with sudo, creating 2 child processes. vmnet-helper drops privileges
immediately after starting the vmnet interface, and run as the user and
group running minikube.
Stopping the cluster will stop sudo, which will stop the vmnet-helper
process. Deleting the cluster kill both sudo and vmnet-helper by killing
the process group.
This change is not complete, but it is good enough to play with the new
shared network.
Example usage:
1. Install vmnet-helper:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#installation
2. Setup vmnet-helper sudoers rule:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#granting-permission-to-run-vmnet-helper
3. Start 2 clusters with vmnet-shared network:
% minikube start -p c1 --driver vfkit --network vmnet-shared
...
% minikube start -p c2 --driver vfkit --network vmnet-shared
...
% minikube ip -p c1
192.168.105.18
% minikube ip -p c2
192.168.105.19
4. Both cluster can access the other cluster:
% minikube -p c1 ssh -- ping -c 3 192.168.105.19
PING 192.168.105.19 (192.168.105.19): 56 data bytes
64 bytes from 192.168.105.19: seq=0 ttl=64 time=0.621 ms
64 bytes from 192.168.105.19: seq=1 ttl=64 time=0.989 ms
64 bytes from 192.168.105.19: seq=2 ttl=64 time=0.490 ms
--- 192.168.105.19 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.490/0.700/0.989 ms
% minikube -p c2 ssh -- ping -c 3 192.168.105.18
PING 192.168.105.18 (192.168.105.18): 56 data bytes
64 bytes from 192.168.105.18: seq=0 ttl=64 time=0.289 ms
64 bytes from 192.168.105.18: seq=1 ttl=64 time=0.798 ms
64 bytes from 192.168.105.18: seq=2 ttl=64 time=0.993 ms
--- 192.168.105.18 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.289/0.693/0.993 ms
* reason: Remove trailing whitepsace
Trailing whitespace is removed by some editors or displayed as a
warning. Clean up to make it easy to make maintain this file.
* start: Validate vfkit --network option
The vfkit driver supports now `nat` and `vmnet-shared` network options.
The `nat` option provides the best performance and is always available,
so it is the default network option. The `vmnet-shared` option provides
access between machines with lower performance compared to `nat`.
If `vment-shared` option is selected, we verify that vmnet-helper is
available. The check ensure that vmnet-helper is installed and sudoers
configuration allows the current user to run vment-helper without a
password.
If validating vment-helper failed, we return a new NotFoundVmnetHelper
reason pointing to vment-helper installation docs or recommending to use
`nat`. This is based on how we treat missing socket_vmnet for QEMU
driver.
* site: Document vfkit network options
* Add new macOS vfkit driver, like hyperkit and qemu
It uses the new Virtualization.framework from macOS 11,
instead of the older Hypervisor.framework (hvf) in QEMU.
* Start vfkit in a new process group
* vfkit: Add firewall workaround from minikube
* vfkit: Add HostIP implementation for gateway
* vfkit: Add driver to extra-disks supported
* Add documentation for vfkit driver
* vfkit: Add usual integration tests
* Update site/content/en/docs/drivers/vfkit.md
Co-authored-by: Steven Powell <44844360+spowelljr@users.noreply.github.com>
---------
Co-authored-by: Nir Soffer <nsoffer@redhat.com>
Co-authored-by: Medya Ghazizadeh <medyagh@users.noreply.github.com>
Co-authored-by: Steven Powell <44844360+spowelljr@users.noreply.github.com>
This allows running Kubernetes with (CRI-)Dockerd in Rootless Docker:
```
minikube start --rootless --driver=docker --container-runtime=docker
```
It is still recommended to use `--container-runtime=containerd`.
Fix issue 17322
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Most people want to use the pre-built binaries of cri-dockerd,
rather than installing go and building it from the source code.
So don't link to the #build-and-install, but to the Releases.
Once it has real documentation URL, that could be used instead.
Nir Soffer
Nir Soffer
Anthony Holloman
Bingtan Lu
Anders Björklund
Benjamin P. Jung
skoenig
Timothée Ravier
Akihiro Suda
Steven Powell
himalayanZephyr
Juan Martín Loyola
Steven Powell
imjoseangel
Carlos Santana
klaases
klaases
shaunmayo