The notify helpers accept now *run.CommandOptions and use it to check if
we can interact with the user. Modify callers to pass options using
cmd/flags.CommandOptions().
vment.ValidateHelper() accept now *run.CommandOptions and use
options.NonInteractive to check if interaction is allowed. Update
callers to pass options from the minikube command.
Testing non-interactive mode:
% sudo rm /etc/sudoers.d/vmnet-helper
% sudo -k
% out/minikube start -d krunkit --interactive=false
😄 minikube v1.37.0 on Darwin 26.0.1 (arm64)
✨ Using the krunkit (experimental) driver based on user configuration
🤷 Exiting due to PROVIDER_KRUNKIT_NOT_FOUND: The 'krunkit' provider was not found: exit status 1: sudo: a password is required
💡 Suggestion: Install and configure vment-helper
📘 Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/krunkit/
Testing interactive mode:
% out/minikube start -d krunkit
😄 minikube v1.37.0 on Darwin 26.0.1 (arm64)
💡 Unable to run vmnet-helper without a password
To configure vment-helper to run without a password, please check the documentation:
https://github.com/nirs/vmnet-helper/#granting-permission-to-run-vmnet-helper
Password:
✨ Using the krunkit (experimental) driver based on user configuration
👍 Starting "minikube" primary control-plane node in "minikube" cluster
🔥 Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.34.1 on Docker 28.4.0 ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Some drivers need command line options since they need to pass command
line options back to minikube firewall package. The way to pass command
line options to the driver is via the NewDriver function, called by the
registry Loader function.
The registry Loader function is called by machine.LocalClient.Load,
which is part of the limachine API interface, which is not part of
minikube so we cannot change it. We pass the options to
machine.NewAPIClient(), so the client can pass the options to Load().
Some drivers need to validate vment helper in the registry StatusChecker
function, considering the --interactive and --download-only flags. So we
pas the options to the StatusChecker function.
This change create the options in most commands that call
machine.NewAPIClient or registry StatusChecker function and pass the
options down.
This change introduce the basic infrastructure for passing command line
options from the cmd/minikube/cmd package to other packages.
The cmd/flags package provides the CommandOptions() function returning
run.CommandOptions initialized using viper. This package keeps the
constants for command line options (e.g. "interactive") that we want to
share with various packages without accessing global state via viper.
To use options in drivers code, include CommandOptions in the
CommonDriver struct. The options will be initialized from the command
line options when creating a driver.
The basic idea is to create options in the command:
options := flags.CommandOptions()
And pass it to other packages, where code will use:
if options.NonInteractive {
Instead of:
if viper.GetBool("interactive") {
This is type safe and allows reliable parallel testing.
In #20255 we added an option to use a configuration file instead of
interactive mode, but the change broke interactive mode. Current
minikube segfaults on start:
% ./out/minikube addons configure registry-creds
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x8 pc=0x1067603dc]
goroutine 1 [running]:
k8s.io/minikube/cmd/minikube/cmd/config.processRegistryCredsConfig({0x106858a06, 0x8}, 0x0)
/Users/nir/src/minikube/cmd/minikube/cmd/config/configure_registry_creds.go:93 +0x2c
k8s.io/minikube/cmd/minikube/cmd/config.init.func8(0x140001f2b00?, {0x140003a83a0, 0x1, 0x106850650?})
/Users/nir/src/minikube/cmd/minikube/cmd/config/configure.go:69 +0x24c
github.com/spf13/cobra.(*Command).execute(0x10a088d40, {0x140003a8350, 0x1, 0x1})
/Users/nir/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1019 +0x82c
github.com/spf13/cobra.(*Command).ExecuteC(0x10a084880)
/Users/nir/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1148 +0x384
github.com/spf13/cobra.(*Command).Execute(...)
/Users/nir/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1071
k8s.io/minikube/cmd/minikube/cmd.Execute()
/Users/nir/src/minikube/cmd/minikube/cmd/root.go:174 +0x550
main.main()
/Users/nir/src/minikube/cmd/minikube/main.go:95 +0x250
The issue is that loadAddonConfigFile() returns nil if the --config-file
flag is not specified, but the code expects non-nil config, handling
zero value as interactive mode. Fixed by returning zero value config in
this case.
With this change we run the normal interactive flow:
% ./out/minikube addons configure registry-creds
Do you want to enable AWS Elastic Container Registry? [y/n]: n
Do you want to enable Google Container Registry? [y/n]: n
Do you want to enable Docker Registry? [y/n]: y
-- Enter docker registry server url: docker.io
-- Enter docker registry username: nirs
-- Enter docker registry password:
Do you want to enable Azure Container Registry? [y/n]: n
✅ registry-creds was successfully configured
% out/minikube addons enable registry-creds
❗ registry-creds is a 3rd party addon and is not maintained or verified by minikube maintainers, enable at your own risk.
❗ registry-creds does not currently have an associated maintainer.
▪ Using image docker.io/upmcenterprises/registry-creds:1.10
🌟 The 'registry-creds' addon is enabled
Note that this addon does not work on arm64 since we have only amd64
image. The pod fail to start:
% kubectl logs deploy/registry-creds -n kube-system
exec /registry-creds: exec format error
Remove the docker-machine-driver-kvm2 wrapper and use the kvm driver as
internal driver.
To avoid dependency on libvirt shared library on Linux, we build now
with the libvirt_dlopen build tag. This is used only linux to avoid
linking with libvirt shared library. This is not documnted but can be
found in the source.
f7cdeba997/domain.go (L30)
With this we don't need libvirt devel libraries during build, and in
runtime we will fail if libvirt shared library is not installed.
With this change minikube can not be built for linux !amd64 since building
libvirt go binding requires CGO, and it does not work by changing GOARCH.
Replace traditional append-in-loop patterns with modern Go functions slices.Sorted(), slices.Collect(), and maps.Keys()/maps.Values()
Changes made (5 files):
- cmd/minikube/cmd/config/addons_list.go: Use slices.Sorted(maps.Keys())
- cmd/minikube/cmd/version.go: Use slices.Sorted(maps.Keys())
- hack/changelog/changelog.go: Use slices.Collect(maps.Keys())
- pkg/minikube/node/cache.go: Use slices.Collect(maps.Keys())
- pkg/minikube/registry/registry.go: Use slices.Collect(maps.Values())
Files skipped due to complexity:
- pkg/drivers/kic/oci/oci.go
- pkg/drivers/hyperkit/driver.go
- pkg/drivers/kvm/gpu.go
- pkg/drivers/kvm/numa.go [Unrelated slice optimization possible, can be addressed along with other similar code]
- pkg/minikube/tunnel/kic/*
- cmd/minikube/cmd/service.go
- hack/legacy_fill_db/filldb.go
These cases require more sophisticated transformation logic that might be better addressed in separate issue/PR
The --mount-string argument defaults to `/Users` on darwin, and
homedir.Homedir() on other platforms (e.g. $HOME on unix).
This is wrong in many ways:
- `/Users` is not HOME on darwin (the right path is `/Users/$USER`).
Using the default mount we cannot access anything inside the guest in
the user home directory. We can access the special `/Users/Shared`
directory, but this should not be a default mount.
- Mounting the user home directory inside the guest in read-write mode
is a horrible default. This exposes the users private keys in .ssh/ to
the guest, any sensitive files in the user home directory, and allows
the guest to change any file on the host.
- Using the `--mount` option mount the default mount directory silently.
This is unexpected, surprising, and not documented in the minikube
handbook[1].
Example access to user private key from the guest with the default
mount:
$ minikube start --mount
$ minikube ssh cat /minikube-host/.ssh/id_ed25519
-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----
Fixed by removing the default mount directory and changing mount logic
to check for non-empty mount-string instead of the mount flag.
The mount flag is kept for backward compatibility, but its value is
ignored. In the next release we want to use this flag for supporting
multiple mounts.
Example usage before:
minikube start --mount --mount-string ~/models:/mnt/models
Example usage after:
minikube start --mount-string ~/models:/mnt/models
Breaking changes:
User depending the default mount will have to replace the command:
minikube start --mount
With:
minikube start --mount-string $HOME:/minikube-host
[1] https://minikube.sigs.k8s.io/docs/handbook/mount/
Move all files and packages which are not a driver to the drivers/common
package. This helps to understand the structure of the code.
While moving, move the iso_test.iso into testdata/test.iso to separate
code and test data.
While fixing the test iso path fix few bugs in the tests using wrong iso
path. The test failed because the iso path was wrong instead of issue
with source or destination path.
New directory structure:
% tree -L1 pkg/drivers
pkg/drivers
├── common
├── hyperkit
├── kic
├── krunkit
├── kvm
├── none
├── qemu
├── ssh
└── vfkit
* Refactored table rendering codes to support updated tablewriter v1.0.7
This commit makes all necessary code changes to maintain compatibility
with the new tablewriter version. The update includes several required modifications across multiple files.
Files modified:
- cmd/minikube/cmd/config/images.go
- cmd/minikube/cmd/config/addons_list.go
- cmd/minikube/cmd/config/profile_list.go
- hack/benchmark/time-to-k8s/chart.go
- hack/benchmark/time-to-k8s/cpu.go
- pkg/minikube/audit/row.go
- pkg/minikube/machine/cache_images.go
- pkg/minikube/perf/result_manager.go
- pkg/minikube/service/service.go
Now #20878 can be merged.
Addresses #20879
* The tablewriter package v1.0.7 renamed SetHeaders to SetHeader
* updated go mod and table.SetColumnAlignment([]int{0, 0, 0, 0})
* Changed syntax and added vendor to gitignore
* simplified version of tablewriter
* removed vendor
* fix addon_list
* fix images tablewrtier
* bump tablewriter for profile list
* go mod tidy
* bump tablewriter for chart
* bump tablewriter for cpu chart
* bump tablewriter for row
* bump tablewriter cache images
* bump tablewriter
* bump tablewriter
* revert file
* bump tablewritter
* fix unit test
* fix lint issues
* fix the integration test pipe char
* convert all tables to new pipe instead of asci pipe
* convert all tables to new pipe instead of asci pipe
* go mod tidy
* go work sync
---------
Co-authored-by: Medya Ghazizadeh <medya@google.com>
* krunkit: Add krunkit driver
krunkit is a tool to launch configurable virtual machines using the
libkrun platform, optimized for GPU accelerated virtual machines and AI
workloads on Apple silicon.
It is mostly compatible with vfkit; the driver is a simplified copy of
the vfkit driver. Unlike vfkit, krunkit is available only on Apple
silicon.
Changes compared to vfkit driver:
- krunkit requires unix socket for networking, so we must use
vment-helper.
- krunkit does not support HardStop, so we kill it using SIGKILL.
- We must enable vmnet offloading, required for krunkit.
- The code was simplified since vmnet-helper is always used
- Code was cleaned up to use .ResolveStorePath()
- Unused Upgrade() function was removed
- Types and functions that should not be public made private
We require krunkit 0.2.2, supporting --restul-uri=unix://.
* reason: Make vment-helper error driver agnostic
Previously it was used only for vfkit, so we suggested to fallback to
the `nat` network. This advice is not relevant to krunkit or to qemu
(which can also use vmnet-helper).
Change the error to recommend installing vment-helper. We need to think
how we can recommend other networks for vfkit and qemu. Another solution
is to create error for every driver+network combination but this seems
hard to manage.
* hack: Add krunkit integration test
This is the same way that we test vfkit. This test is not running in the
CI.
Issues:
- Need to install and configure vment-helper (requires root).
* site: Add krunkit driver documentation
* Fix KVM driver tests timeouts
Rewrite KVM driver waiting logic for domain start, getting ip address
and shutting domain down. Add more config/state outputs to aid future
debugging.
Bump go/libvirt to v1.11002.0 and set the minimum memory required for
running all tests to 3GB to avoid some really weird system behaviour.
* revert reduction of timelimit for TestCert tests run
* set memory and debug output in TestNoKubernetes tests
* extend kvm waitForStaticIP timeout
* add console log to debug output
* Updating ISO to v1.36.0-1748823857-20852
---------
Co-authored-by: minikube-bot <minikube-bot@google.com>
* fix QF1011: could omit type *os.File from declaration; it will be inferred from the right-hand side
* fix QF1012: Use fmt.Fprintf(x, ...) instead of x.Write(fmt.Sprintf(...))
* fix QF1001: could apply De Morgan's law
* fix QF1003: could use tagged switch
* fix weakCond: suspicious ; nil check may not be enough, check for len (gocritic)
* fix docStub: silencing go lint doc-comment warnings is unadvised
* fix builtinShadow: shadowing of predeclared identifier: error
* fix importShadow: shadow of imported package
* fix nestingReduce: invert if cond, replace body with , move old body after the statement
* useless-break: useless break in case clause (revive)
* Clear the redundant content in golangci.yaml file
* vfkit: Fix help text for --extra-disks
vfkit can use extra disks but we forgot to mention it in the help text.
* vfkit: Remove stale tap
vfkit in cfergeau/crc will no longer be updated now that it’s in the main repo.
* vfkit: Fix status in index
The vfkit driver is preferred now.
* Addon configuration now takes an optional config file to load from
instead of forcing prompts
Fixes#20124
* Using a rarer value for default instead of changeme so it has less chance to interfere with other possible user defaults
* Fixing expected values of test cases
* making addonConfigFile private
* Keeping registry_creds configs together.
* Starting a typed AddonConfig struct so each addon con loads its
configs in a typed way.
* Started configs for RegistryCreds using this convention and moved
registry_creds config processing to its own file
(configure_registry_creds.go)
* Making registry creds structs private and other PR feedback cleanups
* Changing default value back to 'changeme'
* Simplifying duplicate consts
* Adding missed comma
* lint fixes
* Generating docs
* Copyright for new file
* vmnet: Improve --network vmnet-shared validation
Previously we did not check that the helper can run with the
--close-from=4 option, so the command could succeed when incorrect
sudoers configuration. For example a user with liberal NOPASSWD rule,
but without the closefrom_override option.
When the check failed, we log unhelpful log:
libmachine: Failed to run vmnet-helper:
%!w(*exec.ExitError=&{0x14000135e30 [115 117 100 111 58 32 97 ... 101 100 10]})
And we returned a bool, so the caller could not provide a suggestion how
to resolve the issue.
Fix by:
- Rename vment.HelperAvaialble to vment.ValidateHelper
- Return an error describing the issue, including a reason.Kind that can
be used to provide a suggestion for resolving the issue.
- Include the ExitError.Stderr int the error. This includes helpful
error messages from sudo.
- Add new reason.NotConfiguredVmnetHelper error
- Improve log when vment.ValidateHelper() succeeded
Example error flow - vment-helper not installed:
% minikube start --driver vfkit --network vmnet-shared
😄 minikube v1.35.0 on Darwin 15.4.1 (arm64)
✨ Using the vfkit (experimental) driver based on user configuration
🙈 Exiting due to NOT_FOUND_VMNET_HELPER: failed to validate vmnet-shared network:
stat /opt/vmnet-helper/bin/vmnet-helper: no such file or directory
💡 Suggestion:
vmnet-helper was not found on the system, resolve by:
Option 1) Installing vmnet-helper:
https://github.com/nirs/vmnet-helper#installation
Option 2) Using the nat network:
minikube start<no value> --driver vfkit --network nat
I resolved the issue by installing vmnet-helper but I did not configured
the sudoers rule:
% minikube start --driver vfkit --network vmnet-shared
😄 minikube v1.35.0 on Darwin 15.4.1 (arm64)
✨ Using the vfkit (experimental) driver based on user configuration
🙈 Exiting due to NOT_CONFIGURED_VMNET_HELPER: failed to validate vmnet-shared network:
exit status 1: sudo: you are not permitted to use the -C option
💡 Suggestion:
Configure vmnet-helper to run without a password.
Please install a vmnet-helper sudoers rule using these instructions:
https://github.com/nirs/vmnet-helper#granting-permission-to-run-vmnet-helper
After installing the sudoers rule minikube could start.
* vfkit: Use helper --socket instead of --fd
The --fd option avoids the need to manage a bound unix sockets, in
particular the limit on unix socket length. It is also more secure;
only the process inheriting the socket can access the helper. However it
requires the sudo --close-from= option, which may not work for some
users. We don't understand why it does not work, and debugging it is
hard since users are not happy to share their local sudoers
configuration.
Avoid the trouble by switching to the --socket option. In this case we
pass a unix socket path to the helper and vfkit. The helper creates a
bound unix datagram socket in the specified path, and waits until vfkit
connects to the socket. When vfkit connects to the unix socket the
programs are connected in the same way they are connected by passing
file descriptors.
When running minikube we will see 3 new files in the machine directory:
- `vfkit-fb64-7802.sock`: vfkit unix datagram socket
- `vmnet-helper.sock`: vmnet-helper unix datagram socket
- `vmnet-helper.sock.lock`: lockfile for vment-helper socket
The files are deleted when vmnet-helper and vfkit are terminated
gracefully. If they are killed the stale files are replaced on the next
start.
Issues:
- If the path exceeds the limit (104 characters), opening the socket
will fail. We have the sames issue with vfkit management socket.
* vment: Fallback to interactive sudo
If vmnet-helper sudoers rule is not configured or does not work for the
user, maybe because the user has disabled the NOPASSWD option, we used
to fail, recommending to configure vmnet sudoers rule. This does not
help a user that cannot fix the sudoers configuration.
Since we switched to --socket mode, we can work without a sudoers rule.
If we can interact with the user, we fall back to interactive sudo. The
user can enter a password to start the machine.
Example run with --interactive=false:
% minikube start --driver vfkit --network vmnet-shared --interactive=false
😄 minikube v1.35.0 on Darwin 15.4.1 (arm64)
✨ Using the vfkit (experimental) driver based on user configuration
🙈 Exiting due to NOT_CONFIGURED_VMNET_HELPER: failed to validate vmnet-shared network:
exit status 1: sudo: a password is required
💡 Suggestion:
Configure vmnet-helper to run without a password.
Please install a vmnet-helper sudoers rule using these instructions:
https://github.com/nirs/vmnet-helper#granting-permission-to-run-vmnet-helper
Example run with --interactive (default):
% minikube start --driver vfkit --network vmnet-shared
😄 minikube v1.35.0 on Darwin 15.4.1 (arm64)
✨ Using the vfkit (experimental) driver based on user configuration
💡 Unable to run vmnet-helper without a password
To configure vment-helper to run without a password, please check the documentation:
https://github.com/nirs/vmnet-helper/#granting-permission-to-run-vmnet-helper
Password:
👍 Starting "minikube" primary control-plane node in "minikube" cluster
🔥 Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.33.0 on Docker 27.4.0 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
* vfkit: Remove temporary variable
Remove temporary and unneeded mac variable. It is easier to follow the
code when we use d.MACAddress.
* vfkit: Promote state change to INFO level
System state changes should be more visible to make debugging easier.
* vmnet: Add vmnet package
The package manages the vmnet-helper[1] child process, providing
connection to the vmnet network without running the guest as root.
We will use vmnet-helper for the vfkit driver, which does not have a way
to use shared network, when guests can access other guest in the
network. We can use it later with the qemu driver as alternative to
socket_vmnet.
[1] https://github.com/nirs/vmnet-helper
* vfkit: add vmnet-shared network
Add new network option for vfkit "vmnet-shared", connecting vfkit to the
vmnet shared network. Clusters using this network can access other
clusters in the same network, similar to socket_vmnet with QEMU driver.
If network is not specified, we default to the "nat" network, keeping
the previous behavior. If network is "vmnet-shared", the vfkit driver
manages 2 processes: vfkit and vmnet-helper.
Like vfkit, vmnet-helper is started in the background, in a new process
group, so it not terminated if the minikube process group is terminate.
Since vmnet-helper requires root to start the vmnet interface, we start
it with sudo, creating 2 child processes. vmnet-helper drops privileges
immediately after starting the vmnet interface, and run as the user and
group running minikube.
Stopping the cluster will stop sudo, which will stop the vmnet-helper
process. Deleting the cluster kill both sudo and vmnet-helper by killing
the process group.
This change is not complete, but it is good enough to play with the new
shared network.
Example usage:
1. Install vmnet-helper:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#installation
2. Setup vmnet-helper sudoers rule:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#granting-permission-to-run-vmnet-helper
3. Start 2 clusters with vmnet-shared network:
% minikube start -p c1 --driver vfkit --network vmnet-shared
...
% minikube start -p c2 --driver vfkit --network vmnet-shared
...
% minikube ip -p c1
192.168.105.18
% minikube ip -p c2
192.168.105.19
4. Both cluster can access the other cluster:
% minikube -p c1 ssh -- ping -c 3 192.168.105.19
PING 192.168.105.19 (192.168.105.19): 56 data bytes
64 bytes from 192.168.105.19: seq=0 ttl=64 time=0.621 ms
64 bytes from 192.168.105.19: seq=1 ttl=64 time=0.989 ms
64 bytes from 192.168.105.19: seq=2 ttl=64 time=0.490 ms
--- 192.168.105.19 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.490/0.700/0.989 ms
% minikube -p c2 ssh -- ping -c 3 192.168.105.18
PING 192.168.105.18 (192.168.105.18): 56 data bytes
64 bytes from 192.168.105.18: seq=0 ttl=64 time=0.289 ms
64 bytes from 192.168.105.18: seq=1 ttl=64 time=0.798 ms
64 bytes from 192.168.105.18: seq=2 ttl=64 time=0.993 ms
--- 192.168.105.18 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.289/0.693/0.993 ms
* reason: Remove trailing whitepsace
Trailing whitespace is removed by some editors or displayed as a
warning. Clean up to make it easy to make maintain this file.
* start: Validate vfkit --network option
The vfkit driver supports now `nat` and `vmnet-shared` network options.
The `nat` option provides the best performance and is always available,
so it is the default network option. The `vmnet-shared` option provides
access between machines with lower performance compared to `nat`.
If `vment-shared` option is selected, we verify that vmnet-helper is
available. The check ensure that vmnet-helper is installed and sudoers
configuration allows the current user to run vment-helper without a
password.
If validating vment-helper failed, we return a new NotFoundVmnetHelper
reason pointing to vment-helper installation docs or recommending to use
`nat`. This is based on how we treat missing socket_vmnet for QEMU
driver.
* site: Document vfkit network options
* Fix waiting for all kube-system pods having one of specified labels to be Ready
* Fix waiting for all kube-system pods having one of specified labels to be Ready
* Fix waiting for all kube-system pods having one of specified labels to be Ready
Nir Soffer
Nir Soffer
Medya Ghazizadeh
Bob Sira
Medya Ghazizadeh
Shin Jihoon
Kaviraj Sankameeswaran
Predrag Rogic
Aaron Nall
minikube-bot
Jun Sugimoto
Victor Ubahakwe
Divy Singhvi
Imran Pochi
Tian
Sri Panyam
錦南路之花
Steven Powell
fbyrne
Steven Powell