diff --git a/content/en/blog/_posts/2023-03-01-introducing-kwok/index.md b/content/en/blog/_posts/2023-03-01-introducing-kwok/index.md new file mode 100644 index 0000000000..5500a7c5d6 --- /dev/null +++ b/content/en/blog/_posts/2023-03-01-introducing-kwok/index.md @@ -0,0 +1,75 @@ +--- +layout: blog +title: "Introducing KWOK: Kubernetes WithOut Kubelet" +date: 2023-03-01 +slug: introducing-kwok +--- + +**Author:** Shiming Zhang (DaoCloud), Wei Huang (Apple), Yibo Zhuang (Apple) + +KWOK logo + +Have you ever wondered how to set up a cluster of thousands of nodes just in seconds, how to simulate real nodes with a low resource footprint, and how to test your Kubernetes controller at scale without spending much on infrastructure? + +If you answered "yes" to any of these questions, then you might be interested in KWOK, a toolkit that enables you to create a cluster of thousands of nodes in seconds. + +## What is KWOK? + +KWOK stands for Kubernetes WithOut Kubelet. So far, it provides two tools: + +`kwok` +: `kwok` is the cornerstone of this project, responsible for simulating the lifecycle of fake nodes, pods, and other Kubernetes API resources. + +`kwokctl` +: `kwokctl` is a CLI tool designed to streamline the creation and management of clusters, with nodes simulated by `kwok`. + +## Why use KWOK? + +KWOK has several advantages: + +- **Speed**: You can create and delete clusters and nodes almost instantly, without waiting for boot or provisioning. +- **Compatibility**: KWOK works with any tools or clients that are compliant with Kubernetes APIs, such as kubectl, helm, kui, etc. +- **Portability**: KWOK has no specific hardware or software requirements. You can run it using pre-built images, once Docker or Nerdctl is installed. Alternatively, binaries are also available for all platforms and can be easily installed. +- **Flexibility**: You can configure different node types, labels, taints, capacities, conditions, etc., and you can configure different pod behaviors, status, etc. to test different scenarios and edge cases. +- **Performance**: You can simulate thousands of nodes on your laptop without significant consumption of CPU or memory resources. + +## What are the use cases? + +KWOK can be used for various purposes: + +- **Learning**: You can use KWOK to learn about Kubernetes concepts and features without worrying about resource waste or other consequences. +- **Development**: You can use KWOK to develop new features or tools for Kubernetes without accessing to a real cluster or requiring other components. +- **Testing**: + - You can measure how well your application or controller scales with different numbers of nodes and(or) pods. + - You can generate high loads on your cluster by creating many pods or services with different resource requests or limits. + - You can simulate node failures or network partitions by changing node conditions or randomly deleting nodes. + - You can test how your controller interacts with other components or features of Kubernetes by enabling different feature gates or API versions. + +## What are the limitations? + +KWOK is not intended to replace others completely. It has some limitations that you should be aware of: + +- **Functionality**: KWOK is not a kubelet and may exhibit different behaviors in areas such as pod lifecycle management, volume mounting, and device plugins. Its primary function is to simulate updates of node and pod status. +- **Accuracy**: It's important to note that KWOK doesn't accurately reflect the performance or behavior of real nodes under various workloads or environments. Instead, it approximates some behaviors using simple formulas. +- **Security**: KWOK does not enforce any security policies or mechanisms on simulated nodes. It assumes that all requests from the kube-apiserver are authorized and valid. + +## Getting started + +If you are interested in trying out KWOK, please check its [documents] for more details. + +{{< figure src="/blog/2023/03/01/introducing-kwok/manage-clusters.svg" alt="Animation of a terminal showing kwokctl in use" caption="Using kwokctl to manage simulated clusters" >}} + +## Getting Involved + +If you're interested in participating in future discussions or development related to KWOK, there are several ways to get involved: + +- Slack: [#kwok] for general usage discussion, [#kwok-dev] for development discussion. (visit [slack.k8s.io] for a workspace invitation) +- Open Issues/PRs/Discussions in [sigs.k8s.io/kwok] + +We welcome feedback and contributions from anyone who wants to join us in this exciting project. + +[documents]: https://kwok.sigs.k8s.io/ +[sigs.k8s.io/kwok]: https://sigs.k8s.io/kwok/ +[#kwok]: https://kubernetes.slack.com/messages/kwok/ +[#kwok-dev]: https://kubernetes.slack.com/messages/kwok-dev/ +[slack.k8s.io]: https://slack.k8s.io/ diff --git a/content/en/blog/_posts/2023-03-01-introducing-kwok/kwok.svg b/content/en/blog/_posts/2023-03-01-introducing-kwok/kwok.svg new file mode 100644 index 0000000000..50c6fed067 --- /dev/null +++ b/content/en/blog/_posts/2023-03-01-introducing-kwok/kwok.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/content/en/blog/_posts/2023-03-01-introducing-kwok/manage-clusters.svg b/content/en/blog/_posts/2023-03-01-introducing-kwok/manage-clusters.svg new file mode 100644 index 0000000000..12a24197ce --- /dev/null +++ b/content/en/blog/_posts/2023-03-01-introducing-kwok/manage-clusters.svg @@ -0,0 +1 @@ +~/go/src/sigs.k8s.io/kwok$~/go/src/sigs.k8s.io/kwok$#~/go/src/sigs.k8s.io/kwok$#Let's~/go/src/sigs.k8s.io/kwok$#Let'sgetting~/go/src/sigs.k8s.io/kwok$#Let'sgettingstarted~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwith~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkwokctl!~/go/src/sigs.k8s.io/kwok$k~/go/src/sigs.k8s.io/kwok$kw~/go/src/sigs.k8s.io/kwok$kwo~/go/src/sigs.k8s.io/kwok$kwok~/go/src/sigs.k8s.io/kwok$kwokc~/go/src/sigs.k8s.io/kwok$kwokct~/go/src/sigs.k8s.io/kwok$kwokctl~/go/src/sigs.k8s.io/kwok$kwokctl-~/go/src/sigs.k8s.io/kwok$kwokctl--~/go/src/sigs.k8s.io/kwok$kwokctl--n~/go/src/sigs.k8s.io/kwok$kwokctl--na~/go/src/sigs.k8s.io/kwok$kwokctl--nam~/go/src/sigs.k8s.io/kwok$kwokctl--name~/go/src/sigs.k8s.io/kwok$kwokctl--named~/go/src/sigs.k8s.io/kwok$kwokctl--namede~/go/src/sigs.k8s.io/kwok$kwokctl--namedem~/go/src/sigs.k8s.io/kwok$kwokctl--namedemo~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreate~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreateclusterCreatingclustercluster=demoStartingclustercluster=demo[+]Running2/0[+]Running5/3Networkkwok-demoCreated0.0sContainerkwok-demo-etcdSt...0.1sContainerkwok-demo-kube-apiserverCreated0.0sContainerkwok-demo-kwok-controllerCreated0.0sContainerkwok-demo-kube-controller-managerCreated0.0sContainerkwok-demo-kube-schedulerCreated0.0s[+]Running5/6Containerkwok-demo-etcdSt...0.3sContainerkwok-demo-kube-apiserverStarting0.3sContainerkwok-demo-kube-apiserverStarting0.5s[+]Running3/6Containerkwok-demo-kube-apiserverStarted0.5sContainerkwok-demo-kwok-controllerStarting0.6sContainerkwok-demo-kube-controller-managerStarting0.6sContainerkwok-demo-kube-schedulerStarting0.6sContainerkwok-demo-kwok-controllerStarting0.7sContainerkwok-demo-kube-controller-managerStarting0.7sContainerkwok-demo-kube-schedulerStarting0.7s[+]Running6/6Containerkwok-demo-kwok-controllerStarted0.8sContainerkwok-demo-kube-controller-managerStarted0.8sContainerkwok-demo-kube-schedulerStarted0.7sClusteriscreatedcluster=demoelapsed=1sYoucannowuseyourclusterwith:kubectlconfiguse-contextkwok-demoThanksforusingkwok!~/go/src/sigs.k8s.io/kwok$#G~/go/src/sigs.k8s.io/kwok$#Ge~/go/src/sigs.k8s.io/kwok$#Get~/go/src/sigs.k8s.io/kwok$#Geta~/go/src/sigs.k8s.io/kwok$#Getal~/go/src/sigs.k8s.io/kwok$#Getall~/go/src/sigs.k8s.io/kwok$#Getallclusters.~/go/src/sigs.k8s.io/kwok$kwokctlget~/go/src/sigs.k8s.io/kwok$kwokctlgetclustersdemo~/go/src/sigs.k8s.io/kwok$#Switch~/go/src/sigs.k8s.io/kwok$#Switchto~/go/src/sigs.k8s.io/kwok$#Switchtothe~/go/src/sigs.k8s.io/kwok$#Switchtothecluster~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontext~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwith~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithkubectl.~/go/src/sigs.k8s.io/kwok$ku~/go/src/sigs.k8s.io/kwok$kub~/go/src/sigs.k8s.io/kwok$kube~/go/src/sigs.k8s.io/kwok$kubec~/go/src/sigs.k8s.io/kwok$kubect~/go/src/sigs.k8s.io/kwok$kubectl~/go/src/sigs.k8s.io/kwok$kubectlconfig~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-context~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwok-demoSwitchedtocontext"kwok-demo".~/go/src/sigs.k8s.io/kwok$#Create~/go/src/sigs.k8s.io/kwok$#Createa~/go/src/sigs.k8s.io/kwok$#Createanode.~/go/src/sigs.k8s.io/kwok$kubectla~/go/src/sigs.k8s.io/kwok$kubectlap~/go/src/sigs.k8s.io/kwok$kubectlapp~/go/src/sigs.k8s.io/kwok$kubectlappl~/go/src/sigs.k8s.io/kwok$kubectlapply~/go/src/sigs.k8s.io/kwok$kubectlapply-~/go/src/sigs.k8s.io/kwok$kubectlapply-f~/go/src/sigs.k8s.io/kwok$kubectlapply-f.~/go/src/sigs.k8s.io/kwok$kubectlapply-f./~/go/src/sigs.k8s.io/kwok$kubectlapply-f./t~/go/src/sigs.k8s.io/kwok$kubectlapply-f./te~/go/src/sigs.k8s.io/kwok$kubectlapply-f./tes~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/k~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kw~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwo~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/f~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fa~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fak~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-node.yamlnode/fake-nodecreated~/go/src/sigs.k8s.io/kwok$#Getallnodes.~/go/src/sigs.k8s.io/kwok$kubectlg~/go/src/sigs.k8s.io/kwok$kubectlge~/go/src/sigs.k8s.io/kwok$kubectlget~/go/src/sigs.k8s.io/kwok$kubectlgetnodeNAMESTATUSROLESAGEVERSIONfake-nodeReadyagent5sfake~/go/src/sigs.k8s.io/kwok$#Apply~/go/src/sigs.k8s.io/kwok$#Applya~/go/src/sigs.k8s.io/kwok$#Applyadeployment.~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployment.yamldeployment.apps/fake-podcreated~/go/src/sigs.k8s.io/kwok$#Getallpods.~/go/src/sigs.k8s.io/kwok$kubectlgetpodNAMEREADYSTATUSRESTARTSAGEfake-pod-5f58597466-ffq791/1Running05sfake-pod-5f58597466-gl2dc1/1Running05sfake-pod-5f58597466-p6vvw1/1Running05sfake-pod-5f58597466-wk44d1/1Running05sfake-pod-5f58597466-xzwbq1/1Running05s~/go/src/sigs.k8s.io/kwok$#Delete~/go/src/sigs.k8s.io/kwok$#Deletethe~/go/src/sigs.k8s.io/kwok$#Deletethecluster.~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodelete~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeleteclusterStoppingclustercluster=demo[+]Running0/0Containerkwok-demo-kwok-controllerRemoved0.1sContainerkwok-demo-kube-controller-managerStopping0.2sContainerkwok-demo-kube-schedulerStopping0.2s[+]Running3/3Containerkwok-demo-kube-controller-managerRemoved0.2sContainerkwok-demo-kube-schedulerRemoved0.2sContainerkwok-demo-kube-apiserverStopping0.1s[+]Running3/4Containerkwok-demo-kube-apiserverStopping0.2sContainerkwok-demo-kube-apiserverStopping0.7sContainerkwok-demo-kube-apiserverStopping1.0s[+]Running4/4Containerkwok-demo-kube-apiserverRemoved1.3sContainerkwok-demo-etcdSt...0.0s[+]Running5/5Containerkwok-demo-etcdRe...0.1sNetworkkwok-demoRemoving0.0sNetworkkwok-demoRemoved0.0sDeletingclustercluster=demoClusterdeletedcluster=demo~/go/src/sigs.k8s.io/kwok$#That's~/go/src/sigs.k8s.io/kwok$#That'sall,~/go/src/sigs.k8s.io/kwok$#That'sall,enjoy~/go/src/sigs.k8s.io/kwok$#That'sall,enjoyit!~/go/src/sigs.k8s.io/kwok$clear~/go/src/sigs.k8s.io/kwok$#L~/go/src/sigs.k8s.io/kwok$#Le~/go/src/sigs.k8s.io/kwok$#Let~/go/src/sigs.k8s.io/kwok$#Let'~/go/src/sigs.k8s.io/kwok$#Let'sg~/go/src/sigs.k8s.io/kwok$#Let'sge~/go/src/sigs.k8s.io/kwok$#Let'sget~/go/src/sigs.k8s.io/kwok$#Let'sgett~/go/src/sigs.k8s.io/kwok$#Let'sgetti~/go/src/sigs.k8s.io/kwok$#Let'sgettin~/go/src/sigs.k8s.io/kwok$#Let'sgettings~/go/src/sigs.k8s.io/kwok$#Let'sgettingst~/go/src/sigs.k8s.io/kwok$#Let'sgettingsta~/go/src/sigs.k8s.io/kwok$#Let'sgettingstar~/go/src/sigs.k8s.io/kwok$#Let'sgettingstart~/go/src/sigs.k8s.io/kwok$#Let'sgettingstarte~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedw~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwi~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwit~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithk~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkw~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkwo~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkwok~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkwokc~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkwokct~/go/src/sigs.k8s.io/kwok$#Let'sgettingstartedwithkwokctl~/go/src/sigs.k8s.io/kwok$kwokctl--namedemoc~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocr~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocre~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocrea~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreat~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreatec~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreatecl~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreateclu~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreateclus~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreateclust~/go/src/sigs.k8s.io/kwok$kwokctl--namedemocreateclusteNetworkkwok-demoCreated0.0sContainerkwok-demo-etcdCreated0.0sContainerkwok-demo-kube-apiserverC...0.0sContainerkwok-demo-etcdSt...0.2sContainerkwok-demo-kube-apiserverStarting0.4s~/go/src/sigs.k8s.io/kwok$#Getallc~/go/src/sigs.k8s.io/kwok$#Getallcl~/go/src/sigs.k8s.io/kwok$#Getallclu~/go/src/sigs.k8s.io/kwok$#Getallclus~/go/src/sigs.k8s.io/kwok$#Getallclust~/go/src/sigs.k8s.io/kwok$#Getallcluste~/go/src/sigs.k8s.io/kwok$#Getallcluster~/go/src/sigs.k8s.io/kwok$#Getallclusters~/go/src/sigs.k8s.io/kwok$kwokctlg~/go/src/sigs.k8s.io/kwok$kwokctlge~/go/src/sigs.k8s.io/kwok$kwokctlgetc~/go/src/sigs.k8s.io/kwok$kwokctlgetcl~/go/src/sigs.k8s.io/kwok$kwokctlgetclu~/go/src/sigs.k8s.io/kwok$kwokctlgetclus~/go/src/sigs.k8s.io/kwok$kwokctlgetclust~/go/src/sigs.k8s.io/kwok$kwokctlgetcluste~/go/src/sigs.k8s.io/kwok$kwokctlgetcluster~/go/src/sigs.k8s.io/kwok$#S~/go/src/sigs.k8s.io/kwok$#Sw~/go/src/sigs.k8s.io/kwok$#Swi~/go/src/sigs.k8s.io/kwok$#Swit~/go/src/sigs.k8s.io/kwok$#Switc~/go/src/sigs.k8s.io/kwok$#Switcht~/go/src/sigs.k8s.io/kwok$#Switchtot~/go/src/sigs.k8s.io/kwok$#Switchtoth~/go/src/sigs.k8s.io/kwok$#Switchtothec~/go/src/sigs.k8s.io/kwok$#Switchtothecl~/go/src/sigs.k8s.io/kwok$#Switchtotheclu~/go/src/sigs.k8s.io/kwok$#Switchtotheclus~/go/src/sigs.k8s.io/kwok$#Switchtotheclust~/go/src/sigs.k8s.io/kwok$#Switchtothecluste~/go/src/sigs.k8s.io/kwok$#Switchtotheclusterc~/go/src/sigs.k8s.io/kwok$#Switchtotheclusterco~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercon~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercont~/go/src/sigs.k8s.io/kwok$#Switchtotheclusterconte~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontex~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextw~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwi~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwit~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithk~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithku~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithkub~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithkube~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithkubec~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithkubect~/go/src/sigs.k8s.io/kwok$#Switchtotheclustercontextwithkubectl~/go/src/sigs.k8s.io/kwok$kubectlc~/go/src/sigs.k8s.io/kwok$kubectlco~/go/src/sigs.k8s.io/kwok$kubectlcon~/go/src/sigs.k8s.io/kwok$kubectlconf~/go/src/sigs.k8s.io/kwok$kubectlconfi~/go/src/sigs.k8s.io/kwok$kubectlconfigu~/go/src/sigs.k8s.io/kwok$kubectlconfigus~/go/src/sigs.k8s.io/kwok$kubectlconfiguse~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-c~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-co~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-con~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-cont~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-conte~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contex~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextk~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkw~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwo~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwok~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwok-~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwok-d~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwok-de~/go/src/sigs.k8s.io/kwok$kubectlconfiguse-contextkwok-dem~/go/src/sigs.k8s.io/kwok$#C~/go/src/sigs.k8s.io/kwok$#Cr~/go/src/sigs.k8s.io/kwok$#Cre~/go/src/sigs.k8s.io/kwok$#Crea~/go/src/sigs.k8s.io/kwok$#Creat~/go/src/sigs.k8s.io/kwok$#Createan~/go/src/sigs.k8s.io/kwok$#Createano~/go/src/sigs.k8s.io/kwok$#Createanod~/go/src/sigs.k8s.io/kwok$#Createanode~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-n~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-no~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-nod~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-node~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-node.~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-node.y~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-node.ya~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-node.yam~/go/src/sigs.k8s.io/kwok$#Getalln~/go/src/sigs.k8s.io/kwok$#Getallno~/go/src/sigs.k8s.io/kwok$#Getallnod~/go/src/sigs.k8s.io/kwok$#Getallnode~/go/src/sigs.k8s.io/kwok$#Getallnodes~/go/src/sigs.k8s.io/kwok$kubectlgetn~/go/src/sigs.k8s.io/kwok$kubectlgetno~/go/src/sigs.k8s.io/kwok$kubectlgetnod~/go/src/sigs.k8s.io/kwok$#A~/go/src/sigs.k8s.io/kwok$#Ap~/go/src/sigs.k8s.io/kwok$#App~/go/src/sigs.k8s.io/kwok$#Appl~/go/src/sigs.k8s.io/kwok$#Applyad~/go/src/sigs.k8s.io/kwok$#Applyade~/go/src/sigs.k8s.io/kwok$#Applyadep~/go/src/sigs.k8s.io/kwok$#Applyadepl~/go/src/sigs.k8s.io/kwok$#Applyadeplo~/go/src/sigs.k8s.io/kwok$#Applyadeploy~/go/src/sigs.k8s.io/kwok$#Applyadeploym~/go/src/sigs.k8s.io/kwok$#Applyadeployme~/go/src/sigs.k8s.io/kwok$#Applyadeploymen~/go/src/sigs.k8s.io/kwok$#Applyadeployment~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-d~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-de~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-dep~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-depl~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deplo~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deploy~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deploym~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployme~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deploymen~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployment~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployment.~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployment.y~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployment.ya~/go/src/sigs.k8s.io/kwok$kubectlapply-f./test/kwok/fake-deployment.yam~/go/src/sigs.k8s.io/kwok$#Getallp~/go/src/sigs.k8s.io/kwok$#Getallpo~/go/src/sigs.k8s.io/kwok$#Getallpod~/go/src/sigs.k8s.io/kwok$#Getallpods~/go/src/sigs.k8s.io/kwok$kubectlgetp~/go/src/sigs.k8s.io/kwok$kubectlgetpofake-pod-5f58597466-p6vvw~/go/src/sigs.k8s.io/kwok$#D~/go/src/sigs.k8s.io/kwok$#De~/go/src/sigs.k8s.io/kwok$#Del~/go/src/sigs.k8s.io/kwok$#Dele~/go/src/sigs.k8s.io/kwok$#Delet~/go/src/sigs.k8s.io/kwok$#Deletet~/go/src/sigs.k8s.io/kwok$#Deleteth~/go/src/sigs.k8s.io/kwok$#Deletethec~/go/src/sigs.k8s.io/kwok$#Deletethecl~/go/src/sigs.k8s.io/kwok$#Deletetheclu~/go/src/sigs.k8s.io/kwok$#Deletetheclus~/go/src/sigs.k8s.io/kwok$#Deletetheclust~/go/src/sigs.k8s.io/kwok$#Deletethecluste~/go/src/sigs.k8s.io/kwok$#Deletethecluster~/go/src/sigs.k8s.io/kwok$kwokctl--namedemod~/go/src/sigs.k8s.io/kwok$kwokctl--namedemode~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodel~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodele~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodelet~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeletec~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeletecl~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeleteclu~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeleteclus~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeleteclust~/go/src/sigs.k8s.io/kwok$kwokctl--namedemodeleteclusteContainerkwok-demo-kwok-controllerStopping0.1sContainerkwok-demo-kube-controller-managerStopping0.1sContainerkwok-demo-kube-schedulerStopping0.1s[+]Running1/3Containerkwok-demo-kube-apiserverStopping0.3sContainerkwok-demo-kube-apiserverStopping0.4sContainerkwok-demo-kube-apiserverStopping0.5sContainerkwok-demo-kube-apiserverStopping0.6sContainerkwok-demo-kube-apiserverStopping0.8sContainerkwok-demo-kube-apiserverStopping0.9sContainerkwok-demo-kube-apiserverStopping1.1sContainerkwok-demo-kube-apiserverStopping1.2sContainerkwok-demo-kube-apiserverStopping1.3s~/go/src/sigs.k8s.io/kwok$#T~/go/src/sigs.k8s.io/kwok$#Th~/go/src/sigs.k8s.io/kwok$#Tha~/go/src/sigs.k8s.io/kwok$#That~/go/src/sigs.k8s.io/kwok$#That'~/go/src/sigs.k8s.io/kwok$#That'sa~/go/src/sigs.k8s.io/kwok$#That'sal~/go/src/sigs.k8s.io/kwok$#That'sall~/go/src/sigs.k8s.io/kwok$#That'sall,e~/go/src/sigs.k8s.io/kwok$#That'sall,en~/go/src/sigs.k8s.io/kwok$#That'sall,enj~/go/src/sigs.k8s.io/kwok$#That'sall,enjo~/go/src/sigs.k8s.io/kwok$#That'sall,enjoyi~/go/src/sigs.k8s.io/kwok$#That'sall,enjoyit~/go/src/sigs.k8s.io/kwok$c~/go/src/sigs.k8s.io/kwok$cl~/go/src/sigs.k8s.io/kwok$cle~/go/src/sigs.k8s.io/kwok$clea \ No newline at end of file