* Add process package
Current code contains multiple implementations for managing a process
using pids, with various issues:
- Some are unsafe, terminating a process by pid without validating that
the pid belongs to the right process. Some use unclear
- Using unclear terms like checkPid() (what does it mean?)
- Some are missing tests
Let's clean up the mess by introducing a process package. The package
provides:
- process.WritePidfile(): write a pid to file
- process.ReadPidfile(): read pid from file
- process.Exists(): tells if process matching pid and name exists
- process.Terminate() terminates a process matching pid and name
- process.Kil() kill a process matching pid and name
The library is tested on linux, darwin, and windows. On windows we don't
have a standard way to terminate a process gracefully, so
process.Terminate() is the same as process.Kill().
I want to use this package in vfkit and the new vment package, and later
we can use it for qemu, hyperkit, and other code using managing
processes with pids.
* vfkit: Use process pidfile helpers
- Simplify GetState() using process.ReadPidfile()
- Simplify Start() using process.WritePidfile()
* vfkit: Simpler and more robust GetState()
GetState() had several issues:
- When accessing vfkit HTTP API, we handled only "running",
"VirtualMachineStateRunning", "stopped", and
"VirtualMachineStateStopped", but there are other 10 possible states,
which we handled as state.None, when vfkit is running and need to be
stopped. This can lead to wrong handling in the caller.
- When handling "stopped" and "VirtualMachineStateStopped" we returned
state.Stopped, but did not remove the pidfile. This can cause
termination of unrelated process or reporting wrong status when the
pid is reused.
- Accessing the HTTP API will fail after we stop or kill it. This
cause GetState() to fail when the process is actually stopped, which
can lead to unnecessary retries and long delays (#20503).
- When retuning state.None during Remove(), we use tried to do graceful
shutdown which does not make sense in minikube delete flow, and is not
consistent with state.Running handling.
Accessing vfkit API to check for state does not add much value for our
used case, checking if the vfkit process is running, and it is not
reliable.
Fix all the issues by not using the HTTP API in GetState(), and use only
the process state. We still use the API for stopping and killing vfkit
to do graceful shutdown. This also simplifies Remove(), since we need to
handle only the state.Running state.
With this change we consider vfkit as stopped only when the process does
not exist, which takes about 3 seconds after the state is reported as
"stopped".
Example stop flow:
I0309 18:15:40.260249 18857 main.go:141] libmachine: Stopping "minikube"...
I0309 18:15:40.263225 18857 main.go:141] libmachine: set state: {State:Stop}
I0309 18:15:46.266902 18857 main.go:141] libmachine: Machine "minikube" was stopped.
I0309 18:15:46.267122 18857 stop.go:75] duration metric: took 6.127761459s to stop
Example delete flow:
I0309 17:00:49.483078 18127 out.go:177] * Deleting "minikube" in vfkit ...
I0309 17:00:49.499252 18127 main.go:141] libmachine: set state: {State:HardStop}
I0309 17:00:49.569938 18127 lock.go:35] WriteFile acquiring /Users/nir/.kube/config: ...
I0309 17:00:49.573977 18127 out.go:177] * Removed all traces of the "minikube" cluster.
* vfkit: Use process.Exists()
Previously we did not check the process name when checking a pid from a
pidfile. If the pidfile became state we would assume that vfkit is
running and try to stop it via the HTTP API, which would never succeed.
Now we detect stale pidfile and remove it.
If removing the stale pidfile fails, we don't want to fail the operation
since we know that vfkit is not running. We log the failure to aid
debugging of stale pidfile.
* vfkit: More robust Stop()
If setting vfkit state to "Stop" fails, we used to return an error.
Retrying the operation may never succeed.
Fix by falling back to terminating vfkit using a signal. This terminates
vfkit immediately similar to HardStop[1].
We can still fail if the pidfile is corrupted but this is unlikely and
requires manual cleanup.
In the case when we are sure the vfkit process does not exist, we remove
the pidfile immediately, avoiding leftover pidfile if the caller does
not call GetState() after Stop().
[1] https://github.com/crc-org/vfkit/issues/284
* vfkit: More robust Kill()
We know that setting the state to `HardStop` typically fails:
I0309 19:19:42.378591 21795 out.go:177] 🔥 Deleting "minikube" in vfkit ...
W0309 19:19:42.397472 21795 delete.go:106] remove failed, will retry: kill: Post "http://_/vm/state": EOF
This may lead to unnecessary retries and delays. Fix by falling back to
sending a SIGKILL signal.
Example delete flow when setting vfkit state fails:
I0309 20:07:41.688259 25540 out.go:177] 🔥 Deleting "minikube" in vfkit ...
I0309 20:07:41.712017 25540 main.go:141] libmachine: Failed to set vfkit state to 'HardStop': Post "http://_/vm/state": EOF
We use `HardStop` which seems to do a forced shutdown instead of
graceful shutdown, and to make things worse, always fails with:
I0309 15:00:42.452986 13723 stop.go:66] stop err: Post "http://_/vm/state": EOF
Which leads to unneeded retry and pointless backup attempt timing out
after 135 seconds because vkfit was stopped.
With this change we do a graceful shutdown, and the time to stop the
cluster decreased from 135 seconds to 3 seconds (45 times faster).
Example stop log:
I0309 15:34:33.104429 14440 main.go:141] libmachine: Stopping "minikube"...
I0309 15:34:33.105225 14440 main.go:141] libmachine: get state: {State:VirtualMachineStateRunning}
I0309 15:34:33.105799 14440 main.go:141] libmachine: set state: {State:Stop}
I0309 15:34:33.106099 14440 main.go:141] libmachine: get state: {State:VirtualMachineStateRunning}
I0309 15:34:36.109380 14440 main.go:141] libmachine: Machine "minikube" was stopped.
minikube-bot
Kubernetes Prow Robot
dependabot[bot]
Sylvester Carolan
cdw8431
Medya Ghazizadeh
Nir Soffer
Aaina Lohia
Tian
Daniel Pepuho