(Addons) Create a new command in every retry
Whenever a `kubectl apply` fails while enabling an addon, it is retried with exponential backoff. The command (type `*exec.Cmd`) that this retry function runs in created outside the function - which means that it is reused on every retry. This is a problem because `exec.Cmd` (https://godoc.org/github.com/pkg/exec#Cmd) states that "... Cmd cannot be reused after calling its Run or Start methods." This retry is a common case due to, say, a CRD and its resource being present in the same YAML file of the addon which causes a race condition where the resource is created before its CRD is created in the cluster - this race is fixed by subsequent retries. I've noticed this in the dashboard and the ambassador addon. Due to the above mentioned bug, minikube throws errors like `exec: already started` in every retry and the retry is never successful, manifests are never deployed and addon creation errors out. Related to #8138 #8119 #8372pull/8792/head
parent
aaebd88251
commit
f42f79e56d
|
|
@ -250,11 +250,9 @@ func enableOrDisableAddonInternal(cc *config.ClusterConfig, addon *assets.Addon,
|
|||
}
|
||||
}
|
||||
|
||||
command := kubectlCommand(cc, deployFiles, enable)
|
||||
|
||||
// Retry, because sometimes we race against an apiserver restart
|
||||
apply := func() error {
|
||||
_, err := cmd.RunCmd(command)
|
||||
_, err := cmd.RunCmd(kubectlCommand(cc, deployFiles, enable))
|
||||
if err != nil {
|
||||
glog.Warningf("apply failed, will retry: %v", err)
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue