* vfkit: Log serial console to file
To make debugging easier, add virtio-serial device logging serial
console to file:
~/.minikube/machines/NAME/serial.log
To enable logging, we need to enable the console in the kernel command
line, since we still use direct kernel boot.
Example log:
% cat /Users/nir/.minikube/machines/vfkit/vfkig.log
[ 0.896094] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 0.897186] loop: module loaded
[ 0.897670] virtio_blk virtio2: [vda] 840488 512-byte logical blocks (430 MB/410 MiB)
[ 0.897733] vda: detected capacity change from 0 to 430329856
[ 0.898460] virtio_blk virtio3: [vdb] 40960000 512-byte logical blocks (21.0 GB/19.5 GiB)
[ 0.898533] vdb: detected capacity change from 0 to 20971520000
...
[ 1.794714] systemd[1]: Detected virtualization vm-other.
[ 1.794752] systemd[1]: Detected architecture arm64.
Welcome to Buildroot 2025.02!
[ 1.794944] systemd[1]: Hostname set to <minikube>.
[ 1.795011] systemd[1]: Initializing machine ID from random generator.
...
[ OK ] Started Container Runtime Interface for OCI (CRI-O).
[ OK ] Reached target Multi-User System.
Welcome to minikube
vfkit login: [ 6.681578] systemd-ssh-generator[630]: Binding SSH to AF_UNIX socket /run/ssh-unix-local/socket.
* vfkit: Use EFI bootloader
With the fixed iso, we can simplify the driver using the EFI bootloader
option[1] instead of the legacy and deprecated --kernel, --kernel-cmdline,
and --initrd options[2].
Example run:
% minikube start -p vfkit --driver vfkit --container-runtime containerd --network vmnet-shared
😄 [vfkit] minikube v1.36.0 on Darwin 15.5 (arm64)
✨ Using the vfkit driver based on user configuration
👍 Starting "vfkit" primary control-plane node in "vfkit" cluster
🔥 Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
📦 Preparing Kubernetes v1.33.1 on containerd 1.7.23 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "vfkit" cluster and "default" namespace by default
Comparing direct kernel boot and --bootloader efi shows that it is little bit faster and boot time is more consistent.
% hyperfine -r 10 -C "minikube delete" \
"vfkit-efi/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes" \
"vfkit-direct/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes"
Benchmark 1: vfkit-efi/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes
Time (mean ± σ): 10.205 s ± 0.656 s [User: 0.381 s, System: 0.266 s]
Range (min … max): 9.106 s … 11.254 s 10 runs
Benchmark 2: vfkit-direct/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes
Time (mean ± σ): 10.933 s ± 1.616 s [User: 0.402 s, System: 0.406 s]
Range (min … max): 9.155 s … 14.168 s 10 runs
Summary
vfkit-efi/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes ran
1.07 ± 0.17 times faster than vfkit-direct/out/minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --no-kubernetes
[1] https://github.com/crc-org/vfkit/blob/main/doc/usage.md#efi-bootloader
[2] https://github.com/crc-org/vfkit/blob/main/doc/usage.md#deprecated-options
* docs: Update vfkit driver documentation
- Separate vfkit requirements and vmnet-shared requirements
- Update minimal macOS version required for --bootloader efi
- Simplify vfkit upgrade, it is available in brew now
Testing shows that we need changes changes:
- x86_64 cpu
- Ubuntu 22.04
- docker is required even if building without docker
- python2 instead of python
- genisoimage (for mkisofs)
- Installing Go manually (Ubuntu 22.04 have only Go 1.18)
- Target should be minikube-iso-aarch64 or minikube-iso-x86_64. Using
arm64 and amd64 fails.
I also cleaned up a little bit the formatting to make it easier to
maintain (one package per line).
Tested building:
- minikube-iso-aarch64
- minikube-iso-x86_64
I did not test the built iso images.
* vfkit: Fix help text for --extra-disks
vfkit can use extra disks but we forgot to mention it in the help text.
* vfkit: Remove stale tap
vfkit in cfergeau/crc will no longer be updated now that it’s in the main repo.
* vfkit: Fix status in index
The vfkit driver is preferred now.
* Kicbase/ISO: Update cni-plugins from v1.6.2 to v1.7.1
* Updating kicbase image to v0.0.46-1747341282-20771
* Updating ISO to v1.35.0-1747341198-20771
* Kicbase/ISO: Update cri-dockerd from v0.3.15 to v0.4.0
* Updating kicbase image to v0.0.46-1747166185-20747
* Updating ISO to v1.35.0-1747160120-20747
* Addon configuration now takes an optional config file to load from
instead of forcing prompts
Fixes#20124
* Using a rarer value for default instead of changeme so it has less chance to interfere with other possible user defaults
* Fixing expected values of test cases
* making addonConfigFile private
* Keeping registry_creds configs together.
* Starting a typed AddonConfig struct so each addon con loads its
configs in a typed way.
* Started configs for RegistryCreds using this convention and moved
registry_creds config processing to its own file
(configure_registry_creds.go)
* Making registry creds structs private and other PR feedback cleanups
* Changing default value back to 'changeme'
* Simplifying duplicate consts
* Adding missed comma
* lint fixes
* Generating docs
* Copyright for new file
* vfkit: Remove temporary variable
Remove temporary and unneeded mac variable. It is easier to follow the
code when we use d.MACAddress.
* vfkit: Promote state change to INFO level
System state changes should be more visible to make debugging easier.
* vmnet: Add vmnet package
The package manages the vmnet-helper[1] child process, providing
connection to the vmnet network without running the guest as root.
We will use vmnet-helper for the vfkit driver, which does not have a way
to use shared network, when guests can access other guest in the
network. We can use it later with the qemu driver as alternative to
socket_vmnet.
[1] https://github.com/nirs/vmnet-helper
* vfkit: add vmnet-shared network
Add new network option for vfkit "vmnet-shared", connecting vfkit to the
vmnet shared network. Clusters using this network can access other
clusters in the same network, similar to socket_vmnet with QEMU driver.
If network is not specified, we default to the "nat" network, keeping
the previous behavior. If network is "vmnet-shared", the vfkit driver
manages 2 processes: vfkit and vmnet-helper.
Like vfkit, vmnet-helper is started in the background, in a new process
group, so it not terminated if the minikube process group is terminate.
Since vmnet-helper requires root to start the vmnet interface, we start
it with sudo, creating 2 child processes. vmnet-helper drops privileges
immediately after starting the vmnet interface, and run as the user and
group running minikube.
Stopping the cluster will stop sudo, which will stop the vmnet-helper
process. Deleting the cluster kill both sudo and vmnet-helper by killing
the process group.
This change is not complete, but it is good enough to play with the new
shared network.
Example usage:
1. Install vmnet-helper:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#installation
2. Setup vmnet-helper sudoers rule:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#granting-permission-to-run-vmnet-helper
3. Start 2 clusters with vmnet-shared network:
% minikube start -p c1 --driver vfkit --network vmnet-shared
...
% minikube start -p c2 --driver vfkit --network vmnet-shared
...
% minikube ip -p c1
192.168.105.18
% minikube ip -p c2
192.168.105.19
4. Both cluster can access the other cluster:
% minikube -p c1 ssh -- ping -c 3 192.168.105.19
PING 192.168.105.19 (192.168.105.19): 56 data bytes
64 bytes from 192.168.105.19: seq=0 ttl=64 time=0.621 ms
64 bytes from 192.168.105.19: seq=1 ttl=64 time=0.989 ms
64 bytes from 192.168.105.19: seq=2 ttl=64 time=0.490 ms
--- 192.168.105.19 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.490/0.700/0.989 ms
% minikube -p c2 ssh -- ping -c 3 192.168.105.18
PING 192.168.105.18 (192.168.105.18): 56 data bytes
64 bytes from 192.168.105.18: seq=0 ttl=64 time=0.289 ms
64 bytes from 192.168.105.18: seq=1 ttl=64 time=0.798 ms
64 bytes from 192.168.105.18: seq=2 ttl=64 time=0.993 ms
--- 192.168.105.18 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.289/0.693/0.993 ms
* reason: Remove trailing whitepsace
Trailing whitespace is removed by some editors or displayed as a
warning. Clean up to make it easy to make maintain this file.
* start: Validate vfkit --network option
The vfkit driver supports now `nat` and `vmnet-shared` network options.
The `nat` option provides the best performance and is always available,
so it is the default network option. The `vmnet-shared` option provides
access between machines with lower performance compared to `nat`.
If `vment-shared` option is selected, we verify that vmnet-helper is
available. The check ensure that vmnet-helper is installed and sudoers
configuration allows the current user to run vment-helper without a
password.
If validating vment-helper failed, we return a new NotFoundVmnetHelper
reason pointing to vment-helper installation docs or recommending to use
`nat`. This is based on how we treat missing socket_vmnet for QEMU
driver.
* site: Document vfkit network options
* Kicbase/ISO: Update cni-plugins from v1.6.1 to v1.6.2
* Updating kicbase image to v0.0.45-1736763277-20236
* Updating ISO to v1.34.0-1736762773-20236
Nir Soffer
minikube-bot
Nir Soffer
Cosmic Oppai
Anthony Holloman
Sri Panyam
錦南路之花
锦南路之花
Hello World