From dc75539de039b769e97add6144a56493cda20f62 Mon Sep 17 00:00:00 2001 From: flo-oss Date: Thu, 21 Jan 2021 19:42:24 -0800 Subject: [PATCH 001/218] added German translation for pod overview --- .../de/docs/concepts/workloads/pods/_index.md | 372 ++++++++++++++++++ 1 file changed, 372 insertions(+) create mode 100644 content/de/docs/concepts/workloads/pods/_index.md diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md new file mode 100644 index 0000000000..d2a2009f8c --- /dev/null +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -0,0 +1,372 @@ +--- +reviewers: +- erictune +title: Pods +content_type: concept +weight: 10 +no_list: true +card: + name: concepts + weight: 60 +--- + + + +_Pods_ sind die kleinsten installierbaren Softwareeinheiten, die in Kubernetes +erstellt und verwaltet werden können. + +Ein _Pod_ (übersetzt Gruppe/Schote, wie z. B. eine Gruppe von Walen oder eine +Erbsenschote) ist eine Gruppe von einem oder mehreren +{{< glossary_tooltip text="Containern" term_id="container" >}} mit gemeinsam +genutzten Speicher- und Netzwerkressourcen und einer Spezifikation für die +Ausführung der Container. Die Ressourcen eines Pods befinden sich immer auf dem +gleichen (virtuellen) Server, werden gemeinsam geplant und in einem +gemeinsamen Kontext ausgeführt. Ein Pod modelliert einen anwendungsspezifischen +"logischen Server": Er enthält eine oder mehrere containerisierte Anwendungen +, die + relativ stark +voneinander abhängen. In Nicht-Cloud-Kontexten sind Anwendungen, die auf +demselben physischen oder virtuellen Server ausgeführt werden, vergleichbar zu +Cloud-Anwendungen, die auf demselben logischen Server ausgeführt werden. + +Ein Pod kann neben Anwendungs-Containern auch sogenannte +[Initialisierungs-Container](/docs/concepts/workloads/pods/init-containers/) +enthalten, die beim Starten des Pods ausgeführt werden. +Es können auch +kurzlebige/[ephemere Container](/docs/concepts/workloads/pods/ephemeral-containers/) +zum Debuggen gestartet werden, wenn dies der Cluster anbietet. + + + +## Was ist ein Pod? + +{{< note >}} +Obwohl Kubernetes abgesehen von [Docker](https://www.docker.com/) auch andere +{{}} unterstützt, ist Docker am bekanntesten und + es ist hilfreich, Pods mit der Terminologie von Docker zu beschreiben. +{{}} + +Der gemeinsame Kontext eines Pods besteht aus einer Reihe von Linux-Namespaces, +Cgroups und möglicherweise andere Aspekten der Isolation, also die gleichen +Dinge, die einen Dockercontainer isolieren. Innerhalb des Kontexts eines Pods +können die einzelnen Anwendungen weitere Unterisolierungen haben. + +Im Sinne von Docker-Konzepten ähnelt ein Pod einer Gruppe von Docker-Containern, +die gemeinsame Namespaces und Dateisystem-Volumes nutzen. + +## Pods verwenden + +Normalerweise müssen keine Pods erzeugt werden, auch keine Singleton-Pods. +Stattdessen werden sie mit Workload-Ressourcen wie {{}} oder {{}} erzeugt. Für Pods, die von einem Systemzustand +abhängen, erwägen Sie die Nutzung von {{}}-Ressourcen. + +Pods in einem Kubernetes-Cluster werden hauptsächlich auf zwei Arten verwendet: + +* **Pods, die einen einzelnen Container ausführen**. Das +"Ein-Container-per-Pod"-Modell ist der häufigste Kubernetes-Anwendungsfall. In +diesem Fall können Sie sich einen Pod als einen Behälter vorstellen, der einen +einzelnen Container enthält; Kubernetes verwaltet die Pods anstatt die +Container direkt zu verwalten. +* **Pods, in denen mehrere Container ausgeführt werden, die zusammenarbeiten +müssen**. Wenn eine Softwareanwendung aus co-lokaliserten Containern besteht, +die sich gemeinsame Ressourcen teilen und stark voneinander abhängen, kann ein +Pod die Container verkapseln. +Diese Container bilden eine einzelne zusammenhängende +Serviceeinheit, z. B. ein Container, der Daten in einem gemeinsam genutzten +Volume öffentlich verfügbar macht, während ein separater _Sidecar_-Container +die Daten aktualisiert. Der Pod fasst die Container, die Speicherressourcen +und eine kurzlebiges Netzwerk-Identität als eine Einheit zusammen. + +{{}} +Das Gruppieren mehrerer gemeinsam lokalisierter und gemeinsam verwalteter +Container in einem einzigen Pod ist ein relativ fortgeschrittener +Anwendungsfall. Sie sollten diese Architektur nur in bestimmten Fällen +verwenden, wenn Ihre Container stark voneinander abhängen. +{{}} + +Jeder Pod sollte eine einzelne Instanz einer gegebenen Anwendung ausführen. Wenn +Sie Ihre Anwendung horizontal skalieren wollen (also mehr Instanzen auszuführen +und dadurch mehr Gesamtressourcen bereitstellen), sollten Sie mehrere Pods +verwenden, +einen für jede Instanz. In Kubernetes wird dies typischerweise als Replikation +bezeichnet. +Replizierte Pods werden normalerweise als eine Gruppe von Workload-Ressource +und {{}} erstellt +und verwaltet. + +Die Seite [Pods und Controller](#pods-and-controllers) beschreibt, wie Kubernetes +Workload-Ressourcen und deren Controller verwendet, um Anwendungen zu skalieren +und zu heilen. + +### Wie Pods mehrere Container verwalten + +Pods unterstützen mehrere kooperierende Prozesse (als Container), die eine +zusammenhängende Serviceeinheit bilden. Kubernetes plant und stellt automatisch +sicher, dass sich die Container in einem Pod auf demselben physischen oder +virtuellen Server im Cluster befinden. Die Container können Ressourcen und +Abhängigkeiten gemeinsam nutzen, miteinander kommunizieren und +ferner koordinieren wann und wie sie beendet werden. + +Zum Beispiel könnten Sie einen Container haben, der als Webserver für Dateien in +einem gemeinsamen Volume arbeitet. Und ein separater "Sidecar" -Container +aktualisiert die Daten von einer externen Datenquelle, siehe folgenden +Abbildung: + +{{< figure src="/images/docs/pod.svg" alt="example pod diagram" width="50%" >}} + +Einige Pods haben sowohl {{}} als auch {{}}. +Initialisierungs-Container werden gestartet und angehalten bevor die +Anwendungs-Container gestartet werden. + +Pods stellen standardmäßig zwei Arten von gemeinsam Ressourcen für die +enthaltenen Container bereit: +[Netzwerk](#pod-networking) und [Speicher](#pod-storage).. + + +## Mit Pods arbeiten + +Sie werden selten einzelne Pods direkt in Kubernetes erstellen, selbst +Singleton-Pods. Das liegt daran, dass Pods als relativ kurzlebige +Einweg-Einheiten konzipiert sind. Wann Ein Pod erstellt wird (entweder direkt +von Ihnen oder indirekt von einem +{{}}), wird die +Ausführung auf einem {{}} in Ihrem Cluster +geplant. Der Pod bleibt auf diesem (virtuellen) Server, bis entweder der Pod die +Ausführung beendet hat, das Pod-Objekt gelöscht wird, der Pod aufgrund +mangelnder Ressourcen *evakuiert* wird oder oder der Node ausfällt. + +{{}} +Das Neustarten eines Containers in einem Pod sollte nicht mit dem Neustarten +eines Pods verwechselt werden. Ein Pod ist kein Prozess, sondern eine Umgebung +zur Ausführung von Containern. Ein Pod bleibt bestehen bis er gelöscht wird. +{{}} + +Stellen Sie beim Erstellen des Manifests für ein Pod-Objekt sicher, dass der +angegebene Name ein gültiger +[DNS-Subdomain-Name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names) +ist. + +### Pods und Controller + +Mit Workload-Ressourcen können Sie mehrere Pods erstellen und verwalten. Ein +Controller für die Ressource kümmert sich um Replikation, Roll-Out sowie +automatische Heilung im Fall von Podfehlern. Wenn beispielsweise ein Node +ausfällt, bemerkt ein Controller, dass die Pods auf dem Node nicht mehr laufen +und plant die Ausführung eines Ersatzpods auf einem funktionierenden Node. +Hier sind einige Beispiele für Workload-Ressourcen, die einen oder mehrere Pods +verwalten: + +* {{< glossary_tooltip text="Deployment" term_id="deployment" >}} +* {{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}} +* {{< glossary_tooltip text="DaemonSet" term_id="daemonset" >}} + +### Podvorlagen + +Controller für +{{}}-Ressourcen +erstellen Pods von einer _Podvorlage_ und verwalten diese Pods für sie. + +Podvorlagen sind Spezifikationen zum Erstellen von Pods und sind in +Workload-Ressourcen enthalten wie z. B. +[Deployments](/docs/concepts/workloads/controllers/deployment/), +[Jobs](/docs/concepts/workloads/controllers/job/), and +[DaemonSets](/docs/concepts/workloads/controllers/daemonset/). + +Jeder Controller für eine Workload-Ressource verwendet die Podvorlage innerhalb +des Workload-Objektes, um Pods zu erzeugen. Die Podvorlage ist Bestandteil des +gewünschten Zustands der Workload-Ressource, mit der Sie Ihre Anwendung +ausgeführt haben. + +Das folgende Beispiel ist ein Manifest für einen einfachen Job mit einer +`Vorlage`, die einen Container startet. Der Container in diesem Pod druckt +eine Nachricht und pausiert dann. + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: hello +spec: + template: + # This is the pod template + spec: + containers: + - name: hello + image: busybox + command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600'] + restartPolicy: OnFailure + # The pod template ends here +``` +Das Ändern der Podvorlage oder der Wechsel zu einer neuen Podvorlage hat keine +direkten Auswirkungen auf bereits existierende Pods. Wenn Sie die Podvorlage für +eine Workload-Ressource ändern, dann muss diese Ressource die Ersatz-Pods +erstellen, welche die aktualisierte Vorlage verwenden. + +Beispielsweise stellt der StatefulSet-Controller sicher, dass für jedes +StatefulSet-Objekt die ausgeführten Pods mit der aktueller Podvorlage +übereinstimmen. Wenn Sie das StatefulSet bearbeiten und die Vorlage ändern, +beginnt das StatefulSet mit der Erstellung neuer Pods basierend auf der +aktualisierten Vorlage. Schließlich werden alle alten Pods durch neue Pods +ersetzt, und das Update ist abgeschlossen. + +Jede Workload-Ressource implementiert eigenen Regeln für die Umsetzung von +Änderungen der Podvorlage. Wenn Sie mehr über StatefulSet erfahren möchten, +lesen Sie +[Update strategy](/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets) +im Tutorial StatefulSet Basics. + + +Auf Nodes beobachtet oder verwaltet das +{{< glossary_tooltip term_id="kubelet" text="Kubelet" >}} +nicht direkt die Details zu Podvorlagen und Updates. Diese Details sind +abstrahiert. Die Abstraktion und Trennung von Aufgaben vereinfacht die +Systemsemantik und ermöglicht so das Verhalten des Clusters zu ändern ohne +vorhandenen Code zu ändern. + +## Pod Update und Austausch + +Wie im vorherigen Abschnitt erwähnt, erstellt der Controller neue Pods basierend +auf der aktualisierten Vorlage, wenn die Podvorlage für eine Workload-Ressource +geändert wird anstatt die vorhandenen Pods zu aktualisieren oder zu patchen. + +Kubernetes hindert Sie nicht daran, Pods direkt zu verwalten. Es ist möglich, +einige Felder eines laufenden Pods zu aktualisieren. Allerdings haben +Pod-Aktualisierungsvorgänge wie zum Beispiel +[`patch`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#patch-pod-v1-core), +und +[`replace`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#replace-pod-v1-core) +einige Einschränkungen: + +- Die meisten Metadaten zu einem Pod sind unveränderlich. Zum Beispiel können + Sie nicht die Felder `namespace`, `name`, `uid`, oder `creationTimestamp` + ändern. Das `generation`-Feld ist eindeutig. Es werden nur Aktualisierungen + akzeptiert, die den Wert des Feldes inkrementieren. +- Wenn das Feld `metadata.deletionTimestamp` gesetzt ist, kann kein neuer + Eintrag zur Liste `metadata.finalizers` hinzugefügt werden. +- Pod-Updates dürfen keine Felder ändern, die Ausnahmen sind + `spec.containers [*].image`, + `spec.initContainers [*].image`,` spec.activeDeadlineSeconds` oder + `spec.tolerations`. Für `spec.tolerations` können Sie nur neue Einträge + hinzufügen. +- Für `spec.activeDeadlineSeconds` sind nur zwei Änderungen erlaubt: + + 1. ungesetztes Feld in eine positive Zahl + 1. positive Zahl in eine kleinere positive Zahl, die nicht negativ ist + +## Gemeinsame Nutzung von Ressourcen und Kommunikation + +Pods ermöglichen den Datenaustausch und die Kommunikation zwischen den +Containern, die im Pod enthalten sind. + +### Datenspeicherung in Pods {#pod-storage} + +Ein Pod kann eine Reihe von gemeinsam genutzten Speicher- +{{}} spezifizieren. Alle +Container im Pod können auf die gemeinsamen Volumes zugreifen und dadurch Daten +austauschen. Volumes ermöglichen auch, dass Daten ohne Verlust gespeichert +werden, falls einer der Container neu gestartet werden muss. +Im Kapitel [Datenspeicherung](/docs/concepts/storage/) finden Sie weitere +Informationen, wie Kubernetes gemeinsam genutzten Speicher implementiert und Pods +zur Verfügung stellt. + +### Pod-Netzwerk + +Jedem Pod wird für jede Adressenfamilie eine eindeutige IP-Adresse zugewiesen. +Jeder Container in einem Pod nutzt den gemeinsamen Netzwerk-Namespace, +einschließlich der IP-Adresse und der Ports. In einem Pod (und **nur** dann) +können die Container, die zum Pod gehören, über `localhost` miteinander +kommunizieren. Wenn Container in einem Pod mit Entitäten *außerhalb des Pods* +kommunizieren, müssen sie koordinieren, wie die gemeinsam genutzten +Netzwerkressourcen (z. B. Ports) verwenden werden. Innerhalb eines Pods teilen +sich Container eine IP-Adresse und einen Reihe von Ports und können sich +gegenseitig über `localhost` finden. Die Container in einem Pod können auch die +üblichen Kommunikationsverfahren zwischen Prozessen nutzen, wie z. B. +SystemV-Semaphoren oder POSIX Shared Memory. Container in verschiedenen Pods +haben unterschiedliche IP-Adressen und können nicht per IPC ohne +[spezielle Konfiguration](/docs/concepts/policy/pod-security-policy/) +kommunizieren. Container, die mit einem Container in einem anderen Pod +interagieren möchten, müssen IP Netzwerke verwenden. + +Für die Container innerhalb eines Pods stimmt der "hostname" mit dem +konfigurierten `Namen` des Pods überein. Mehr dazu im Kapitel +[Netzwerke](/docs/concepts/cluster-administration/networking/). + +## Privilegierter Modus für Container + +Jeder Container in einem Pod kann den privilegierten Modus aktivieren, indem +das Flag `privileged` im +[Sicherheitskontext](/docs/tasks/configure-pod-container/security-context/) +der Container-Spezifikation verwendet wird. +Dies ist nützlich für Container, die Verwaltungsfunktionen des Betriebssystems +verwenden möchten, z. B. das Manipulieren des Netzwerk-Stacks oder den Zugriff +auf Hardware. Prozesse innerhalb eines privilegierten Containers erhalten fast +die gleichen Rechte wie sie Prozesse außerhalb eines Containers zur Verfügung +stehen. + +{{}} +Ihre +{{}} +muss das Konzept eines privilegierten Containers unterstützen, damit diese +Einstellung relevant ist. +{{}} + + +## Statische Pods + +_Statische Pods_ werden direkt vom Kubelet-Daemon auf einem bestimmten Node +verwaltet ohne dass sie vom +{{}} überwacht +werden. +. +Die meisten Pods werden von der Control Plane verwaltet (z. B. +{{< glossary_tooltip text="Deployment" term_id="deployment" >}}). Aber für +statische Pods überwacht das Kubelet jeden statischen Pod direkt (und startet +ihn neu, wenn er ausfällt). + +Statische Pods sind immer an ein {{}} auf +einem bestimmten Node gebunden. Der Hauptanwendungsfall für statische Pods +besteht darin, eine selbst gehostete Steuerebene auszuführen. Mit anderen +Worten: Das Kubelet dient zur Überwachung der einzelnen +[control plane components](/docs/concepts/overview/components/#control-plane-components). + +Das Kubelet versucht automatisch auf dem Kubernetes API-Server für jeden +statischen Pod einen spiegelbildlichen Pod +(im Englischen {{}}) +zu erstellen. +Das bedeutet, dass die auf einem Node ausgeführten Pods auf dem API-Server +sichtbar sind jedoch von dort nicht gesteuert werden können. + +## {{% heading "whatsnext" %}} + +* Erfahren Sie mehr über den + [Lebenszyklus eines Pods](/docs/concepts/workloads/pods/pod-lifecycle/). +* Erfahren Sie mehr über [RuntimeClass](/docs/concepts/containers/runtime-class/) + und wie Sie damit verschiedene Pods mit unterschiedlichen + Container-Laufzeitumgebung konfigurieren können. +* Lesen sie mehr über + [Restriktionen für die Verteilung von Pods](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). +* Lesen sie mehr über sogenannte + [Pod-Disruption-Budgets](/docs/concepts/workloads/pods/disruptions/) + und wie Sie diese verwenden können, um die Verfügbarkeit von Anwendungen bei + Störungen zu verwalten. Die + [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) + -Objektdefinition beschreibt das Objekt im Detail. +* [The Distributed System Toolkit: Patterns for Composite Containers](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns) + erläutert allgemeine Layouts für Pods mit mehr als einem Container. + +Um den Hintergrund zu verstehen, warum Kubernetes eine gemeinsame Pod-API in +andere Ressourcen, wie z. B. +{{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} +oder {{< glossary_tooltip text="Deployments" term_id="deployment" >}} einbindet, +können sie Artikel zu früheren Technologien lesen, unter anderem: + * [Aurora](https://aurora.apache.org/documentation/latest/reference/configuration/#job-schema) + * [Borg](https://research.google.com/pubs/pub43438.html) + * [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html) + * [Omega](https://research.google/pubs/pub41684/) + * [Tupperware](https://engineering.fb.com/data-center-engineering/tupperware/). \ No newline at end of file From f213353b3028d6fc8099b7232346321a5f63ed66 Mon Sep 17 00:00:00 2001 From: flo-oss Date: Thu, 21 Jan 2021 20:45:13 -0800 Subject: [PATCH 002/218] added German translation for pod overview page --- .../de/docs/concepts/workloads/pods/_index.md | 49 ++++++++++--------- 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md index d2a2009f8c..b1aec877c9 100644 --- a/content/de/docs/concepts/workloads/pods/_index.md +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -12,7 +12,7 @@ card: -_Pods_ sind die kleinsten installierbaren Softwareeinheiten, die in Kubernetes +_Pods_ sind die kleinsten einsetzbaren Einheiten, die in Kubernetes erstellt und verwaltet werden können. Ein _Pod_ (übersetzt Gruppe/Schote, wie z. B. eine Gruppe von Walen oder eine @@ -42,13 +42,13 @@ zum Debuggen gestartet werden, wenn dies der Cluster anbietet. {{< note >}} Obwohl Kubernetes abgesehen von [Docker](https://www.docker.com/) auch andere -{{}} unterstützt, ist Docker am bekanntesten und es ist hilfreich, Pods mit der Terminologie von Docker zu beschreiben. {{}} Der gemeinsame Kontext eines Pods besteht aus einer Reihe von Linux-Namespaces, -Cgroups und möglicherweise andere Aspekten der Isolation, also die gleichen +Cgroups und möglicherweise anderen Aspekten der Isolation, also die gleichen Dinge, die einen Dockercontainer isolieren. Innerhalb des Kontexts eines Pods können die einzelnen Anwendungen weitere Unterisolierungen haben. @@ -89,18 +89,19 @@ verwenden, wenn Ihre Container stark voneinander abhängen. {{}} Jeder Pod sollte eine einzelne Instanz einer gegebenen Anwendung ausführen. Wenn -Sie Ihre Anwendung horizontal skalieren wollen (also mehr Instanzen auszuführen +Sie Ihre Anwendung horizontal skalieren wollen (um mehr Instanzen auszuführen und dadurch mehr Gesamtressourcen bereitstellen), sollten Sie mehrere Pods verwenden, einen für jede Instanz. In Kubernetes wird dies typischerweise als Replikation bezeichnet. -Replizierte Pods werden normalerweise als eine Gruppe von Workload-Ressource -und {{}} erstellt +Replizierte Pods werden normalerweise als eine Gruppe durch eine +Workload-Ressource und deren +{{}} erstellt und verwaltet. -Die Seite [Pods und Controller](#pods-and-controllers) beschreibt, wie Kubernetes -Workload-Ressourcen und deren Controller verwendet, um Anwendungen zu skalieren -und zu heilen. +Die Seite [Pods und Controller](#pods-and-controllers) beschreibt, wie +Kubernetes Workload-Ressourcen und deren Controller verwendet, um Anwendungen +zu skalieren und zu heilen. ### Wie Pods mehrere Container verwalten @@ -121,12 +122,12 @@ Abbildung: Einige Pods haben sowohl {{}} als auch {{}}. -Initialisierungs-Container werden gestartet und angehalten bevor die +Initialisierungs-Container werden gestartet und beendet bevor die Anwendungs-Container gestartet werden. Pods stellen standardmäßig zwei Arten von gemeinsam Ressourcen für die enthaltenen Container bereit: -[Netzwerk](#pod-networking) und [Speicher](#pod-storage).. +[Netzwerk](#pod-networking) und [Speicher](#pod-storage). ## Mit Pods arbeiten @@ -179,7 +180,7 @@ Workload-Ressourcen enthalten wie z. B. [DaemonSets](/docs/concepts/workloads/controllers/daemonset/). Jeder Controller für eine Workload-Ressource verwendet die Podvorlage innerhalb -des Workload-Objektes, um Pods zu erzeugen. Die Podvorlage ist Bestandteil des +des Workload-Objektes, um Pods zu erzeugen. Die Podvorlage ist Teil des gewünschten Zustands der Workload-Ressource, mit der Sie Ihre Anwendung ausgeführt haben. @@ -217,8 +218,8 @@ ersetzt, und das Update ist abgeschlossen. Jede Workload-Ressource implementiert eigenen Regeln für die Umsetzung von Änderungen der Podvorlage. Wenn Sie mehr über StatefulSet erfahren möchten, -lesen Sie -[Update strategy](/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets) +lesen Sie die Seite +[Update-Strategien](/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets) im Tutorial StatefulSet Basics. @@ -243,9 +244,9 @@ und [`replace`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#replace-pod-v1-core) einige Einschränkungen: -- Die meisten Metadaten zu einem Pod sind unveränderlich. Zum Beispiel können +- Die meisten Metadaten zu einem Pod können nicht verändert werden. Zum Beispiel können Sie nicht die Felder `namespace`, `name`, `uid`, oder `creationTimestamp` - ändern. Das `generation`-Feld ist eindeutig. Es werden nur Aktualisierungen + ändern. Das `generation`-Feld muss eindeutig sein. Es werden nur Aktualisierungen akzeptiert, die den Wert des Feldes inkrementieren. - Wenn das Feld `metadata.deletionTimestamp` gesetzt ist, kann kein neuer Eintrag zur Liste `metadata.finalizers` hinzugefügt werden. @@ -272,8 +273,8 @@ Container im Pod können auf die gemeinsamen Volumes zugreifen und dadurch Daten austauschen. Volumes ermöglichen auch, dass Daten ohne Verlust gespeichert werden, falls einer der Container neu gestartet werden muss. Im Kapitel [Datenspeicherung](/docs/concepts/storage/) finden Sie weitere -Informationen, wie Kubernetes gemeinsam genutzten Speicher implementiert und Pods -zur Verfügung stellt. +Informationen, wie Kubernetes gemeinsam genutzten Speicher implementiert und +Pods zur Verfügung stellt. ### Pod-Netzwerk @@ -284,10 +285,10 @@ können die Container, die zum Pod gehören, über `localhost` miteinander kommunizieren. Wenn Container in einem Pod mit Entitäten *außerhalb des Pods* kommunizieren, müssen sie koordinieren, wie die gemeinsam genutzten Netzwerkressourcen (z. B. Ports) verwenden werden. Innerhalb eines Pods teilen -sich Container eine IP-Adresse und einen Reihe von Ports und können sich +sich Container eine IP-Adresse und eine Reihe von Ports und können sich gegenseitig über `localhost` finden. Die Container in einem Pod können auch die üblichen Kommunikationsverfahren zwischen Prozessen nutzen, wie z. B. -SystemV-Semaphoren oder POSIX Shared Memory. Container in verschiedenen Pods +SystemV-Semaphoren oder "POSIX Shared Memory". Container in verschiedenen Pods haben unterschiedliche IP-Adressen und können nicht per IPC ohne [spezielle Konfiguration](/docs/concepts/policy/pod-security-policy/) kommunizieren. Container, die mit einem Container in einem anderen Pod @@ -306,7 +307,7 @@ der Container-Spezifikation verwendet wird. Dies ist nützlich für Container, die Verwaltungsfunktionen des Betriebssystems verwenden möchten, z. B. das Manipulieren des Netzwerk-Stacks oder den Zugriff auf Hardware. Prozesse innerhalb eines privilegierten Containers erhalten fast -die gleichen Rechte wie sie Prozesse außerhalb eines Containers zur Verfügung +die gleichen Rechte wie sie Prozessen außerhalb eines Containers zur Verfügung stehen. {{}} @@ -324,7 +325,7 @@ verwaltet ohne dass sie vom {{}} überwacht werden. . -Die meisten Pods werden von der Control Plane verwaltet (z. B. +Die meisten Pods werden von der Kontrollebene verwaltet (z. B. {{< glossary_tooltip text="Deployment" term_id="deployment" >}}). Aber für statische Pods überwacht das Kubelet jeden statischen Pod direkt (und startet ihn neu, wenn er ausfällt). @@ -333,7 +334,7 @@ Statische Pods sind immer an ein {{}} auf einem bestimmten Node gebunden. Der Hauptanwendungsfall für statische Pods besteht darin, eine selbst gehostete Steuerebene auszuführen. Mit anderen Worten: Das Kubelet dient zur Überwachung der einzelnen -[control plane components](/docs/concepts/overview/components/#control-plane-components). +[Komponenten der Kontrollebene](/docs/concepts/overview/components/#control-plane-components). Das Kubelet versucht automatisch auf dem Kubernetes API-Server für jeden statischen Pod einen spiegelbildlichen Pod @@ -348,7 +349,7 @@ sichtbar sind jedoch von dort nicht gesteuert werden können. [Lebenszyklus eines Pods](/docs/concepts/workloads/pods/pod-lifecycle/). * Erfahren Sie mehr über [RuntimeClass](/docs/concepts/containers/runtime-class/) und wie Sie damit verschiedene Pods mit unterschiedlichen - Container-Laufzeitumgebung konfigurieren können. + Container-Laufzeitumgebungen konfigurieren können. * Lesen sie mehr über [Restriktionen für die Verteilung von Pods](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). * Lesen sie mehr über sogenannte From 90cc7f10d4413ebbb1a33f0624138fb862e9d864 Mon Sep 17 00:00:00 2001 From: flo-oss Date: Tue, 2 Feb 2021 22:20:15 -0800 Subject: [PATCH 003/218] changes suggested by code reviewer --- .../de/docs/concepts/workloads/pods/_index.md | 67 +++++++++---------- 1 file changed, 32 insertions(+), 35 deletions(-) diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md index b1aec877c9..4080f460f9 100644 --- a/content/de/docs/concepts/workloads/pods/_index.md +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -1,6 +1,4 @@ --- -reviewers: -- erictune title: Pods content_type: concept weight: 10 @@ -22,10 +20,9 @@ genutzten Speicher- und Netzwerkressourcen und einer Spezifikation für die Ausführung der Container. Die Ressourcen eines Pods befinden sich immer auf dem gleichen (virtuellen) Server, werden gemeinsam geplant und in einem gemeinsamen Kontext ausgeführt. Ein Pod modelliert einen anwendungsspezifischen -"logischen Server": Er enthält eine oder mehrere containerisierte Anwendungen -, die - relativ stark -voneinander abhängen. In Nicht-Cloud-Kontexten sind Anwendungen, die auf +"logischen Server": Er enthält eine oder mehrere containerisierte Anwendungen, +die relativ stark voneinander abhängen. +In Nicht-Cloud-Kontexten sind Anwendungen, die auf demselben physischen oder virtuellen Server ausgeführt werden, vergleichbar zu Cloud-Anwendungen, die auf demselben logischen Server ausgeführt werden. @@ -42,10 +39,10 @@ zum Debuggen gestartet werden, wenn dies der Cluster anbietet. {{< note >}} Obwohl Kubernetes abgesehen von [Docker](https://www.docker.com/) auch andere -{{}} unterstützt, ist Docker am bekanntesten und +{{}} unterstützt, ist Docker am bekanntesten und es ist hilfreich, Pods mit der Terminologie von Docker zu beschreiben. -{{}} +{{< /note >}} Der gemeinsame Kontext eines Pods besteht aus einer Reihe von Linux-Namespaces, Cgroups und möglicherweise anderen Aspekten der Isolation, also die gleichen @@ -59,10 +56,10 @@ die gemeinsame Namespaces und Dateisystem-Volumes nutzen. Normalerweise müssen keine Pods erzeugt werden, auch keine Singleton-Pods. Stattdessen werden sie mit Workload-Ressourcen wie {{}} oder {{}} erzeugt. Für Pods, die von einem Systemzustand -abhängen, erwägen Sie die Nutzung von {{}}-Ressourcen. +text="Deployment" term_id="deployment">}} oder {{}} erzeugt. Für Pods, die von einem Systemzustand +abhängen, erwägen Sie die Nutzung von {{}}-Ressourcen. Pods in einem Kubernetes-Cluster werden hauptsächlich auf zwei Arten verwendet: @@ -81,12 +78,12 @@ Volume öffentlich verfügbar macht, während ein separater _Sidecar_-Container die Daten aktualisiert. Der Pod fasst die Container, die Speicherressourcen und eine kurzlebiges Netzwerk-Identität als eine Einheit zusammen. -{{}} +{{< note >}} Das Gruppieren mehrerer gemeinsam lokalisierter und gemeinsam verwalteter Container in einem einzigen Pod ist ein relativ fortgeschrittener Anwendungsfall. Sie sollten diese Architektur nur in bestimmten Fällen verwenden, wenn Ihre Container stark voneinander abhängen. -{{}} +{{< /note >}} Jeder Pod sollte eine einzelne Instanz einer gegebenen Anwendung ausführen. Wenn Sie Ihre Anwendung horizontal skalieren wollen (um mehr Instanzen auszuführen @@ -96,10 +93,10 @@ einen für jede Instanz. In Kubernetes wird dies typischerweise als Replikation bezeichnet. Replizierte Pods werden normalerweise als eine Gruppe durch eine Workload-Ressource und deren -{{}} erstellt +{{}} erstellt und verwaltet. -Die Seite [Pods und Controller](#pods-and-controllers) beschreibt, wie +Der Abschnitt [Pods und Controller](#pods-und-controller) beschreibt, wie Kubernetes Workload-Ressourcen und deren Controller verwendet, um Anwendungen zu skalieren und zu heilen. @@ -117,17 +114,17 @@ einem gemeinsamen Volume arbeitet. Und ein separater "Sidecar" -Container aktualisiert die Daten von einer externen Datenquelle, siehe folgenden Abbildung: -{{< figure src="/images/docs/pod.svg" alt="example pod diagram" width="50%" >}} +{{< figure src="/images/docs/pod.svg" alt="Pod-Beispieldiagramm" width="50%" >}} -Einige Pods haben sowohl {{}} als auch {{}}. +Einige Pods haben sowohl {{}} als auch {{}}. Initialisierungs-Container werden gestartet und beendet bevor die Anwendungs-Container gestartet werden. Pods stellen standardmäßig zwei Arten von gemeinsam Ressourcen für die enthaltenen Container bereit: -[Netzwerk](#pod-networking) und [Speicher](#pod-storage). +[Netzwerk](#pod-netzwerk) und [Speicher](#datenspeicherung-in-pods). ## Mit Pods arbeiten @@ -136,17 +133,17 @@ Sie werden selten einzelne Pods direkt in Kubernetes erstellen, selbst Singleton-Pods. Das liegt daran, dass Pods als relativ kurzlebige Einweg-Einheiten konzipiert sind. Wann Ein Pod erstellt wird (entweder direkt von Ihnen oder indirekt von einem -{{}}), wird die -Ausführung auf einem {{}} in Ihrem Cluster +{{}}), wird die +Ausführung auf einem {{}} in Ihrem Cluster geplant. Der Pod bleibt auf diesem (virtuellen) Server, bis entweder der Pod die Ausführung beendet hat, das Pod-Objekt gelöscht wird, der Pod aufgrund mangelnder Ressourcen *evakuiert* wird oder oder der Node ausfällt. -{{}} +{{< note >}} Das Neustarten eines Containers in einem Pod sollte nicht mit dem Neustarten eines Pods verwechselt werden. Ein Pod ist kein Prozess, sondern eine Umgebung zur Ausführung von Containern. Ein Pod bleibt bestehen bis er gelöscht wird. -{{}} +{{< /note >}} Stellen Sie beim Erstellen des Manifests für ein Pod-Objekt sicher, dass der angegebene Name ein gültiger @@ -170,7 +167,7 @@ verwalten: ### Podvorlagen Controller für -{{}}-Ressourcen +{{}}-Ressourcen erstellen Pods von einer _Podvorlage_ und verwalten diese Pods für sie. Podvorlagen sind Spezifikationen zum Erstellen von Pods und sind in @@ -265,10 +262,10 @@ einige Einschränkungen: Pods ermöglichen den Datenaustausch und die Kommunikation zwischen den Containern, die im Pod enthalten sind. -### Datenspeicherung in Pods {#pod-storage} +### Datenspeicherung in Pods Ein Pod kann eine Reihe von gemeinsam genutzten Speicher- -{{}} spezifizieren. Alle +{{}} spezifizieren. Alle Container im Pod können auf die gemeinsamen Volumes zugreifen und dadurch Daten austauschen. Volumes ermöglichen auch, dass Daten ohne Verlust gespeichert werden, falls einer der Container neu gestartet werden muss. @@ -310,19 +307,19 @@ auf Hardware. Prozesse innerhalb eines privilegierten Containers erhalten fast die gleichen Rechte wie sie Prozessen außerhalb eines Containers zur Verfügung stehen. -{{}} +{{< note >}} Ihre -{{}} +{{}} muss das Konzept eines privilegierten Containers unterstützen, damit diese Einstellung relevant ist. -{{}} +{{< /note >}} ## Statische Pods _Statische Pods_ werden direkt vom Kubelet-Daemon auf einem bestimmten Node verwaltet ohne dass sie vom -{{}} überwacht +{{}} überwacht werden. . Die meisten Pods werden von der Kontrollebene verwaltet (z. B. @@ -330,7 +327,7 @@ Die meisten Pods werden von der Kontrollebene verwaltet (z. B. statische Pods überwacht das Kubelet jeden statischen Pod direkt (und startet ihn neu, wenn er ausfällt). -Statische Pods sind immer an ein {{}} auf +Statische Pods sind immer an ein {{}} auf einem bestimmten Node gebunden. Der Hauptanwendungsfall für statische Pods besteht darin, eine selbst gehostete Steuerebene auszuführen. Mit anderen Worten: Das Kubelet dient zur Überwachung der einzelnen @@ -338,7 +335,7 @@ Worten: Das Kubelet dient zur Überwachung der einzelnen Das Kubelet versucht automatisch auf dem Kubernetes API-Server für jeden statischen Pod einen spiegelbildlichen Pod -(im Englischen {{}}) +(im Englischen: {{}}) zu erstellen. Das bedeutet, dass die auf einem Node ausgeführten Pods auf dem API-Server sichtbar sind jedoch von dort nicht gesteuert werden können. From 99aa654d8be21ab928177a05a70514d586abd030 Mon Sep 17 00:00:00 2001 From: flo-oss Date: Tue, 2 Feb 2021 22:53:55 -0800 Subject: [PATCH 004/218] changes suggested by code reviewer, part 2 --- content/de/docs/concepts/workloads/pods/_index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md index 4080f460f9..837a1f1571 100644 --- a/content/de/docs/concepts/workloads/pods/_index.md +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -192,14 +192,14 @@ metadata: name: hello spec: template: - # This is the pod template + # Dies is the Podvorlage spec: containers: - name: hello image: busybox command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600'] restartPolicy: OnFailure - # The pod template ends here + # Die Podvorlage endet hier ``` Das Ändern der Podvorlage oder der Wechsel zu einer neuen Podvorlage hat keine direkten Auswirkungen auf bereits existierende Pods. Wenn Sie die Podvorlage für @@ -248,8 +248,8 @@ einige Einschränkungen: - Wenn das Feld `metadata.deletionTimestamp` gesetzt ist, kann kein neuer Eintrag zur Liste `metadata.finalizers` hinzugefügt werden. - Pod-Updates dürfen keine Felder ändern, die Ausnahmen sind - `spec.containers [*].image`, - `spec.initContainers [*].image`,` spec.activeDeadlineSeconds` oder + `spec.containers[*].image`, + `spec.initContainers[*].image`,` spec.activeDeadlineSeconds` oder `spec.tolerations`. Für `spec.tolerations` können Sie nur neue Einträge hinzufügen. - Für `spec.activeDeadlineSeconds` sind nur zwei Änderungen erlaubt: From 5b127f07612d45ba0a142370e4e58f9e859c3830 Mon Sep 17 00:00:00 2001 From: flo-oss Date: Tue, 16 Feb 2021 21:07:07 -0800 Subject: [PATCH 005/218] changes suggested by code reviewer (sie -> du, Pod Vorlage, ....) --- .../de/docs/concepts/workloads/pods/_index.md | 97 +++++++++---------- 1 file changed, 48 insertions(+), 49 deletions(-) diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md index 837a1f1571..e6adce49a6 100644 --- a/content/de/docs/concepts/workloads/pods/_index.md +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -57,15 +57,15 @@ die gemeinsame Namespaces und Dateisystem-Volumes nutzen. Normalerweise müssen keine Pods erzeugt werden, auch keine Singleton-Pods. Stattdessen werden sie mit Workload-Ressourcen wie {{}} oder {{}} erzeugt. Für Pods, die von einem Systemzustand -abhängen, erwägen Sie die Nutzung von {{}}-Ressourcen. +text="Job" term_id="job">}} erzeugt. Für Pods, die von einem Systemzustand +abhängen, ist die Nutzung von {{}}-Ressourcen zu erwägen. Pods in einem Kubernetes-Cluster werden hauptsächlich auf zwei Arten verwendet: * **Pods, die einen einzelnen Container ausführen**. Das "Ein-Container-per-Pod"-Modell ist der häufigste Kubernetes-Anwendungsfall. In -diesem Fall können Sie sich einen Pod als einen Behälter vorstellen, der einen +diesem Fall kannst du dir einen einen Pod als einen Behälter vorstellen, der einen einzelnen Container enthält; Kubernetes verwaltet die Pods anstatt die Container direkt zu verwalten. * **Pods, in denen mehrere Container ausgeführt werden, die zusammenarbeiten @@ -81,16 +81,15 @@ und eine kurzlebiges Netzwerk-Identität als eine Einheit zusammen. {{< note >}} Das Gruppieren mehrerer gemeinsam lokalisierter und gemeinsam verwalteter Container in einem einzigen Pod ist ein relativ fortgeschrittener -Anwendungsfall. Sie sollten diese Architektur nur in bestimmten Fällen -verwenden, wenn Ihre Container stark voneinander abhängen. +Anwendungsfall. Du solltest diese Architektur nur in bestimmten Fällen +verwenden, wenn deine Container stark voneinander abhängen. {{< /note >}} Jeder Pod sollte eine einzelne Instanz einer gegebenen Anwendung ausführen. Wenn -Sie Ihre Anwendung horizontal skalieren wollen (um mehr Instanzen auszuführen -und dadurch mehr Gesamtressourcen bereitstellen), sollten Sie mehrere Pods -verwenden, -einen für jede Instanz. In Kubernetes wird dies typischerweise als Replikation -bezeichnet. +du deine Anwendung horizontal skalieren willst (um mehr Instanzen auszuführen +und dadurch mehr Gesamtressourcen bereitstellen), solltest du mehrere Pods +verwenden, einen für jede Instanz. +In Kubernetes wird dies typischerweise als Replikation bezeichnet. Replizierte Pods werden normalerweise als eine Gruppe durch eine Workload-Ressource und deren {{}} erstellt @@ -109,7 +108,7 @@ virtuellen Server im Cluster befinden. Die Container können Ressourcen und Abhängigkeiten gemeinsam nutzen, miteinander kommunizieren und ferner koordinieren wann und wie sie beendet werden. -Zum Beispiel könnten Sie einen Container haben, der als Webserver für Dateien in +Zum Beispiel könntest du einen Container haben, der als Webserver für Dateien in einem gemeinsamen Volume arbeitet. Und ein separater "Sidecar" -Container aktualisiert die Daten von einer externen Datenquelle, siehe folgenden Abbildung: @@ -129,7 +128,7 @@ enthaltenen Container bereit: ## Mit Pods arbeiten -Sie werden selten einzelne Pods direkt in Kubernetes erstellen, selbst +Du wirst selten einzelne Pods direkt in Kubernetes erstellen, selbst Singleton-Pods. Das liegt daran, dass Pods als relativ kurzlebige Einweg-Einheiten konzipiert sind. Wann Ein Pod erstellt wird (entweder direkt von Ihnen oder indirekt von einem @@ -145,16 +144,16 @@ eines Pods verwechselt werden. Ein Pod ist kein Prozess, sondern eine Umgebung zur Ausführung von Containern. Ein Pod bleibt bestehen bis er gelöscht wird. {{< /note >}} -Stellen Sie beim Erstellen des Manifests für ein Pod-Objekt sicher, dass der +Stelle beim Erstellen des Manifests für ein Pod-Objekt sicher, dass der angegebene Name ein gültiger [DNS-Subdomain-Name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names) ist. ### Pods und Controller -Mit Workload-Ressourcen können Sie mehrere Pods erstellen und verwalten. Ein +Mit Workload-Ressourcen kannst du mehrere Pods erstellen und verwalten. Ein Controller für die Ressource kümmert sich um Replikation, Roll-Out sowie -automatische Heilung im Fall von Podfehlern. Wenn beispielsweise ein Node +automatische Wiederherstellung im Fall von versagenden Pods. Wenn beispielsweise ein Node ausfällt, bemerkt ein Controller, dass die Pods auf dem Node nicht mehr laufen und plant die Ausführung eines Ersatzpods auf einem funktionierenden Node. Hier sind einige Beispiele für Workload-Ressourcen, die einen oder mehrere Pods @@ -164,22 +163,22 @@ verwalten: * {{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}} * {{< glossary_tooltip text="DaemonSet" term_id="daemonset" >}} -### Podvorlagen +### Pod Vorlagen Controller für {{}}-Ressourcen -erstellen Pods von einer _Podvorlage_ und verwalten diese Pods für sie. +erstellen Pods von einer _Pod Vorlage_ und verwalten diese Pods für dich. -Podvorlagen sind Spezifikationen zum Erstellen von Pods und sind in +Pod Vorlagen sind Spezifikationen zum Erstellen von Pods und sind in Workload-Ressourcen enthalten wie z. B. [Deployments](/docs/concepts/workloads/controllers/deployment/), [Jobs](/docs/concepts/workloads/controllers/job/), and [DaemonSets](/docs/concepts/workloads/controllers/daemonset/). -Jeder Controller für eine Workload-Ressource verwendet die Podvorlage innerhalb -des Workload-Objektes, um Pods zu erzeugen. Die Podvorlage ist Teil des -gewünschten Zustands der Workload-Ressource, mit der Sie Ihre Anwendung -ausgeführt haben. +Jeder Controller für eine Workload-Ressource verwendet die Pod Vorlage innerhalb +des Workload-Objektes, um Pods zu erzeugen. Die Pod Vorlage ist Teil des +gewünschten Zustands der Workload-Ressource, mit der du deine Anwendung +ausgeführt hast. Das folgende Beispiel ist ein Manifest für einen einfachen Job mit einer `Vorlage`, die einen Container startet. Der Container in diesem Pod druckt @@ -192,37 +191,37 @@ metadata: name: hello spec: template: - # Dies is the Podvorlage + # Dies is the Pod Vorlage spec: containers: - name: hello image: busybox command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600'] restartPolicy: OnFailure - # Die Podvorlage endet hier + # Die Pod Vorlage endet hier ``` -Das Ändern der Podvorlage oder der Wechsel zu einer neuen Podvorlage hat keine -direkten Auswirkungen auf bereits existierende Pods. Wenn Sie die Podvorlage für -eine Workload-Ressource ändern, dann muss diese Ressource die Ersatz-Pods +Das Ändern der Pod Vorlage oder der Wechsel zu einer neuen Pod Vorlage hat keine +direkten Auswirkungen auf bereits existierende Pods. Wenn du die Pod Vorlage für +eine Workload-Ressource änderst, dann muss diese Ressource die Ersatz-Pods erstellen, welche die aktualisierte Vorlage verwenden. Beispielsweise stellt der StatefulSet-Controller sicher, dass für jedes -StatefulSet-Objekt die ausgeführten Pods mit der aktueller Podvorlage -übereinstimmen. Wenn Sie das StatefulSet bearbeiten und die Vorlage ändern, +StatefulSet-Objekt die ausgeführten Pods mit der aktueller Pod Vorlage +übereinstimmen. Wenn du das StatefulSet bearbeitest und die Vorlage änderst, beginnt das StatefulSet mit der Erstellung neuer Pods basierend auf der aktualisierten Vorlage. Schließlich werden alle alten Pods durch neue Pods ersetzt, und das Update ist abgeschlossen. Jede Workload-Ressource implementiert eigenen Regeln für die Umsetzung von -Änderungen der Podvorlage. Wenn Sie mehr über StatefulSet erfahren möchten, -lesen Sie die Seite +Änderungen der Pod Vorlage. Wenn du mehr über StatefulSet erfahren möchtest, +dann lese die Seite [Update-Strategien](/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets) im Tutorial StatefulSet Basics. Auf Nodes beobachtet oder verwaltet das {{< glossary_tooltip term_id="kubelet" text="Kubelet" >}} -nicht direkt die Details zu Podvorlagen und Updates. Diese Details sind +nicht direkt die Details zu Pod Vorlagen und Updates. Diese Details sind abstrahiert. Die Abstraktion und Trennung von Aufgaben vereinfacht die Systemsemantik und ermöglicht so das Verhalten des Clusters zu ändern ohne vorhandenen Code zu ändern. @@ -230,10 +229,10 @@ vorhandenen Code zu ändern. ## Pod Update und Austausch Wie im vorherigen Abschnitt erwähnt, erstellt der Controller neue Pods basierend -auf der aktualisierten Vorlage, wenn die Podvorlage für eine Workload-Ressource +auf der aktualisierten Vorlage, wenn die Pod Vorlage für eine Workload-Ressource geändert wird anstatt die vorhandenen Pods zu aktualisieren oder zu patchen. -Kubernetes hindert Sie nicht daran, Pods direkt zu verwalten. Es ist möglich, +Kubernetes hindert dich nicht daran, Pods direkt zu verwalten. Es ist möglich, einige Felder eines laufenden Pods zu aktualisieren. Allerdings haben Pod-Aktualisierungsvorgänge wie zum Beispiel [`patch`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#patch-pod-v1-core), @@ -241,8 +240,8 @@ und [`replace`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#replace-pod-v1-core) einige Einschränkungen: -- Die meisten Metadaten zu einem Pod können nicht verändert werden. Zum Beispiel können - Sie nicht die Felder `namespace`, `name`, `uid`, oder `creationTimestamp` +- Die meisten Metadaten zu einem Pod können nicht verändert werden. Zum Beispiel kannst + du nicht die Felder `namespace`, `name`, `uid`, oder `creationTimestamp` ändern. Das `generation`-Feld muss eindeutig sein. Es werden nur Aktualisierungen akzeptiert, die den Wert des Feldes inkrementieren. - Wenn das Feld `metadata.deletionTimestamp` gesetzt ist, kann kein neuer @@ -250,7 +249,7 @@ einige Einschränkungen: - Pod-Updates dürfen keine Felder ändern, die Ausnahmen sind `spec.containers[*].image`, `spec.initContainers[*].image`,` spec.activeDeadlineSeconds` oder - `spec.tolerations`. Für `spec.tolerations` können Sie nur neue Einträge + `spec.tolerations`. Für `spec.tolerations` kannnst du nur neue Einträge hinzufügen. - Für `spec.activeDeadlineSeconds` sind nur zwei Änderungen erlaubt: @@ -269,7 +268,7 @@ Ein Pod kann eine Reihe von gemeinsam genutzten Speicher- Container im Pod können auf die gemeinsamen Volumes zugreifen und dadurch Daten austauschen. Volumes ermöglichen auch, dass Daten ohne Verlust gespeichert werden, falls einer der Container neu gestartet werden muss. -Im Kapitel [Datenspeicherung](/docs/concepts/storage/) finden Sie weitere +Im Kapitel [Datenspeicherung](/docs/concepts/storage/) findest du weitere Informationen, wie Kubernetes gemeinsam genutzten Speicher implementiert und Pods zur Verfügung stellt. @@ -321,7 +320,7 @@ _Statische Pods_ werden direkt vom Kubelet-Daemon auf einem bestimmten Node verwaltet ohne dass sie vom {{}} überwacht werden. -. + Die meisten Pods werden von der Kontrollebene verwaltet (z. B. {{< glossary_tooltip text="Deployment" term_id="deployment" >}}). Aber für statische Pods überwacht das Kubelet jeden statischen Pod direkt (und startet @@ -342,16 +341,16 @@ sichtbar sind jedoch von dort nicht gesteuert werden können. ## {{% heading "whatsnext" %}} -* Erfahren Sie mehr über den +* Lerne den [Lebenszyklus eines Pods](/docs/concepts/workloads/pods/pod-lifecycle/). -* Erfahren Sie mehr über [RuntimeClass](/docs/concepts/containers/runtime-class/) - und wie Sie damit verschiedene Pods mit unterschiedlichen - Container-Laufzeitumgebungen konfigurieren können. -* Lesen sie mehr über +* Lerne die [RuntimeClass](/docs/concepts/containers/runtime-class/) + und wie du damit verschiedene Pods mit unterschiedlichen + Container-Laufzeitumgebungen konfigurieren kannst. +* Lese [Restriktionen für die Verteilung von Pods](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). -* Lesen sie mehr über sogenannte - [Pod-Disruption-Budgets](/docs/concepts/workloads/pods/disruptions/) - und wie Sie diese verwenden können, um die Verfügbarkeit von Anwendungen bei +* Lese + [Pod-Disruption-Budget](/docs/concepts/workloads/pods/disruptions/) + und wie du es verwenden kannst, um die Verfügbarkeit von Anwendungen bei Störungen zu verwalten. Die [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) -Objektdefinition beschreibt das Objekt im Detail. @@ -362,7 +361,7 @@ Um den Hintergrund zu verstehen, warum Kubernetes eine gemeinsame Pod-API in andere Ressourcen, wie z. B. {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} oder {{< glossary_tooltip text="Deployments" term_id="deployment" >}} einbindet, -können sie Artikel zu früheren Technologien lesen, unter anderem: +kannst du Artikel zu früheren Technologien lesen, unter anderem: * [Aurora](https://aurora.apache.org/documentation/latest/reference/configuration/#job-schema) * [Borg](https://research.google.com/pubs/pub43438.html) * [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html) From 363545ef9673242e21eacad2c6259a0be803a33c Mon Sep 17 00:00:00 2001 From: flo-oss Date: Mon, 22 Feb 2021 21:35:48 -0800 Subject: [PATCH 006/218] Slightly more creative translation to make it sound less repetitive and less commanding :-) --- content/de/docs/concepts/workloads/pods/_index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/de/docs/concepts/workloads/pods/_index.md b/content/de/docs/concepts/workloads/pods/_index.md index e6adce49a6..956190e6c7 100644 --- a/content/de/docs/concepts/workloads/pods/_index.md +++ b/content/de/docs/concepts/workloads/pods/_index.md @@ -341,12 +341,12 @@ sichtbar sind jedoch von dort nicht gesteuert werden können. ## {{% heading "whatsnext" %}} -* Lerne den +* Verstehe den [Lebenszyklus eines Pods](/docs/concepts/workloads/pods/pod-lifecycle/). -* Lerne die [RuntimeClass](/docs/concepts/containers/runtime-class/) +* Erfahre mehr über [RuntimeClass](/docs/concepts/containers/runtime-class/) und wie du damit verschiedene Pods mit unterschiedlichen Container-Laufzeitumgebungen konfigurieren kannst. -* Lese +* Mehr zum Thema [Restriktionen für die Verteilung von Pods](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). * Lese [Pod-Disruption-Budget](/docs/concepts/workloads/pods/disruptions/) From bc023f265cfe4829e09e74859219dd7fbd65124f Mon Sep 17 00:00:00 2001 From: Richard Li Date: Sat, 27 Feb 2021 07:03:42 -0500 Subject: [PATCH 007/218] add blog post on using kubectl annotate for human service discovery --- .../2021-02-27-annotating-k8s-for-humans.md | 99 +++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md new file mode 100644 index 0000000000..bfc49a431b --- /dev/null +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -0,0 +1,99 @@ +--- +layout: blog +title: 'Annotating Kubernetes Services for Humans' +date: 2021-02-27 +slug: annotating-k8s-for-humans +--- + +**Authors:** Richard Li (Ambassador Labs) + +Have you ever been asked to troubleshoot a failing Kubernetes service and struggled to find basic information about the service such as the source repository and owner? + +One of the problems as Kubernetes applications grow is the proliferation of services. As the number of services grows, developers start to specialize working with specific services. When it comes to troubleshooting, however, developers need to be able to find the source, understand the service and dependencies, and chat with the owning team for any service. + +## Human Service Discovery + +Troubleshooting always begins with information gathering. While much attention has been paid to centralizing machine data (e.g., logs, metrics), much less attention has been given to the human aspect of service discovery. Who owns a particular service? What Slack channel does the team work on? Where is the source for the service? What issues are currently known and being tracked? + +## Kubernetes Annotations + +Kubernetes annotations are designed to solve exactly this problem. Oft-overlooked, Kubernetes annotations are designed to add metadata to Kubernetes objects. The Kubernetes documentation says annotations can “attach arbitrary non-identifying metadata to objects.” This means that annotations should be used for attaching metadata that is external to Kubernetes (i.e., metadata that Kubernetes won’t use to identify objects. As such, annotations can contain any type of data. This is a contrast to labels, which are designed for uses internal to Kubernetes. As such, label structure and values are [constrained](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) so they can be efficiently used by Kubernetes. + + +## Kubernetes Annotations in Action + +Here is an example. Imagine you have a Kubernetes service for quoting, called the quote service. You can do the following: + +``` +$ kubectl annotate service quote a8r.io/owner=”@sally” +``` + +In this example, we’ve just added an annotation called `a8r.io/owner` with the value of @sally. Now, we can use `kubectl describe` to get the information. + +``` +$ kubectl describe svc quote +Name: quote +Namespace: default +Labels: +Annotations: a8r.io/owner: @sally +Selector: app=quote +Type: ClusterIP +IP: 10.109.142.131 +Port: http 80/TCP +TargetPort: 8080/TCP +Endpoints: +Session Affinity: None +Events: +``` + +If you’re practicing GitOps (and you should be!) you’ll want to code these values directly into your Kubernetes manifest, e.g., + +``` +apiVersion: v1 +kind: Service +metadata: + name: quote + annotations: + a8r.io/owner: “@sally” +spec: + ports: + - name: http + port: 80 + targetPort: 8080 + selector: + app: quote +``` + +## A Convention for Annotations + +Adopting a common convention for annotations ensures consistency and understandability. Typically, you’ll want to attach the annotation to the service object, as services are the high-level resource that maps most clearly to a team’s responsibility. Namespacing your annotations is also very important. Here is one set of conventions, documented at [a8r.io](https://a8r.io), and reproduced below: + +| Annotation | Description | +| ------------------------------------------ | ------------------------------------------- | +| `a8r.io/description` | Unstructured text description of the service for humans. | +| `a8r.io/owner` | SSO username (GitHub), email address (linked to GitHub account), or unstructured owner description. | +| `a8r.io/chat` | Slack channel, or link to external chat system | +| `a8r.io/bugs` | Link to external bug tracker | +| `a8r.io/logs` | Link to external log viewer | +| `a8r.io/documentation` | Link to external project documentation | +| `a8r.io/repository` | Link to external VCS repository | +| `a8r.io/support` | Link to external support center | +| `a8r.io/runbook` | Link to external project runbook | +| `a8r.io/incidents` | Link to external incident dashboard | +| `a8r.io/uptime` | Link to external uptime dashboard | +| `a8r.io/performance` | Link to external performance dashboard | +| `a8r.io/dependencies` | Unstructured text describing the service dependencies for humans. | + + +## Visualizing Annotations: Service Catalogs + +As the number of microservices and annotations proliferate, using `kubectl describe` can get tedious. Moreover, kubectl describe requires every developer to have some direct access to the Kubernetes cluster. A [service catalog](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog) presents an internal, developer-oriented view of all services. + +Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include Backstage and Ambassador. [Backstage](https://backstage.io/) is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while [Ambassador](https://www.getambassador.io/products/service-catalog) provides a turnkey, cloud-hosted solution that can be set up quickly. + +## Annotate Your Services Now and Thank Yourself Later +Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Typically, something is on fire in production and you wished you had implemented better metrics and also specified which Slack channel the owning team lived in within your organization. + +There's enormous benefits to building an effective “[version 0](https://www.getambassador.io/learn/kubernetes-glossary/version-0/)” of any service: a “[dancing skeleton](https://containerjournal.com/topics/container-management/dancing-skeleton-apis-and-microservices/)” application with a thin slice of complete functionality that can be deployed to production with a minimal yet effective continuous delivery pipeline. + +Adding service annotations should be an essential part of your “version 0” for all of your services. Add them now, and you’ll thank yourself later. From ecd25dfefb01c707393a579087b39026ab522775 Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:09:13 -0500 Subject: [PATCH 008/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index bfc49a431b..a1969cd9cc 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -17,7 +17,7 @@ Troubleshooting always begins with information gathering. While much attention h ## Kubernetes Annotations -Kubernetes annotations are designed to solve exactly this problem. Oft-overlooked, Kubernetes annotations are designed to add metadata to Kubernetes objects. The Kubernetes documentation says annotations can “attach arbitrary non-identifying metadata to objects.” This means that annotations should be used for attaching metadata that is external to Kubernetes (i.e., metadata that Kubernetes won’t use to identify objects. As such, annotations can contain any type of data. This is a contrast to labels, which are designed for uses internal to Kubernetes. As such, label structure and values are [constrained](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) so they can be efficiently used by Kubernetes. +Kubernetes annotations are designed to solve exactly this problem. Oft-overlooked, Kubernetes annotations are designed to add metadata to Kubernetes objects. The Kubernetes documentation says annotations can “attach arbitrary non-identifying metadata to objects.” This means that annotations should be used for attaching metadata that is external to Kubernetes (i.e., metadata that Kubernetes won’t use to identify objects. As such, annotations can contain any type of data. This is a contrast to labels, which are designed for uses internal to Kubernetes. As such, label structure and values are [constrained](/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) so they can be efficiently used by Kubernetes. ## Kubernetes Annotations in Action From eeb8c0b45ec3972d942c32d791d8d68dbbe5c2fc Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:09:20 -0500 Subject: [PATCH 009/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index a1969cd9cc..9a0c166421 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -87,7 +87,7 @@ Adopting a common convention for annotations ensures consistency and understanda ## Visualizing Annotations: Service Catalogs -As the number of microservices and annotations proliferate, using `kubectl describe` can get tedious. Moreover, kubectl describe requires every developer to have some direct access to the Kubernetes cluster. A [service catalog](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog) presents an internal, developer-oriented view of all services. +As the number of microservices and annotations proliferate, running `kubectl describe` can get tedious. Moreover, using `kubectl describe` requires every developer to have some direct access to the Kubernetes cluster. An [service catalog](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog), in the Ambassador meaning of that term, presents an internal, developer-oriented view of all services. Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include Backstage and Ambassador. [Backstage](https://backstage.io/) is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while [Ambassador](https://www.getambassador.io/products/service-catalog) provides a turnkey, cloud-hosted solution that can be set up quickly. From 69ee6318a5f6d293f2c9a2c3bfd2dfd371af228b Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:10:02 -0500 Subject: [PATCH 010/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index 9a0c166421..ddf0bca364 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -92,7 +92,7 @@ As the number of microservices and annotations proliferate, running `kubectl des Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include Backstage and Ambassador. [Backstage](https://backstage.io/) is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while [Ambassador](https://www.getambassador.io/products/service-catalog) provides a turnkey, cloud-hosted solution that can be set up quickly. ## Annotate Your Services Now and Thank Yourself Later -Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Typically, something is on fire in production and you wished you had implemented better metrics and also specified which Slack channel the owning team lived in within your organization. +Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Don't wait until something is on fire in production to start wishing you had implemented better metrics and also documented how to get in touch with the part of your organization that looks after it. There's enormous benefits to building an effective “[version 0](https://www.getambassador.io/learn/kubernetes-glossary/version-0/)” of any service: a “[dancing skeleton](https://containerjournal.com/topics/container-management/dancing-skeleton-apis-and-microservices/)” application with a thin slice of complete functionality that can be deployed to production with a minimal yet effective continuous delivery pipeline. From dc86d666482105fc0edfe4b392c7ab0d5a54ad2a Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:11:11 -0500 Subject: [PATCH 011/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index ddf0bca364..994601a9f3 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -85,7 +85,7 @@ Adopting a common convention for annotations ensures consistency and understanda | `a8r.io/dependencies` | Unstructured text describing the service dependencies for humans. | -## Visualizing Annotations: Service Catalogs +## Visualizing annotations: service catalogs As the number of microservices and annotations proliferate, running `kubectl describe` can get tedious. Moreover, using `kubectl describe` requires every developer to have some direct access to the Kubernetes cluster. An [service catalog](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog), in the Ambassador meaning of that term, presents an internal, developer-oriented view of all services. From 3188c43241b6be5484b65dd351417ca36b66ef65 Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:11:17 -0500 Subject: [PATCH 012/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index 994601a9f3..1c05fe2410 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -91,7 +91,7 @@ As the number of microservices and annotations proliferate, running `kubectl des Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include Backstage and Ambassador. [Backstage](https://backstage.io/) is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while [Ambassador](https://www.getambassador.io/products/service-catalog) provides a turnkey, cloud-hosted solution that can be set up quickly. -## Annotate Your Services Now and Thank Yourself Later +## Annotate your services now and thank yourself later Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Don't wait until something is on fire in production to start wishing you had implemented better metrics and also documented how to get in touch with the part of your organization that looks after it. There's enormous benefits to building an effective “[version 0](https://www.getambassador.io/learn/kubernetes-glossary/version-0/)” of any service: a “[dancing skeleton](https://containerjournal.com/topics/container-management/dancing-skeleton-apis-and-microservices/)” application with a thin slice of complete functionality that can be deployed to production with a minimal yet effective continuous delivery pipeline. From a381d88fe26590491b5e81652afa18b7651caf9f Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:11:26 -0500 Subject: [PATCH 013/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index 1c05fe2410..87af6e8863 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -11,7 +11,7 @@ Have you ever been asked to troubleshoot a failing Kubernetes service and strugg One of the problems as Kubernetes applications grow is the proliferation of services. As the number of services grows, developers start to specialize working with specific services. When it comes to troubleshooting, however, developers need to be able to find the source, understand the service and dependencies, and chat with the owning team for any service. -## Human Service Discovery +## Human service discovery Troubleshooting always begins with information gathering. While much attention has been paid to centralizing machine data (e.g., logs, metrics), much less attention has been given to the human aspect of service discovery. Who owns a particular service? What Slack channel does the team work on? Where is the source for the service? What issues are currently known and being tracked? From 5b3eedc8c2ccbcb149e535aff6a2ce6d1fb47d37 Mon Sep 17 00:00:00 2001 From: Richard Li Date: Mon, 1 Mar 2021 08:49:44 -0500 Subject: [PATCH 014/218] clarify difference with k8s service catalog Signed-off-by: Richard Li --- .../blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index 87af6e8863..d2c4223713 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -5,7 +5,7 @@ date: 2021-02-27 slug: annotating-k8s-for-humans --- -**Authors:** Richard Li (Ambassador Labs) +**Author:** Richard Li, Ambassador Labs Have you ever been asked to troubleshoot a failing Kubernetes service and struggled to find basic information about the service such as the source repository and owner? @@ -15,7 +15,7 @@ One of the problems as Kubernetes applications grow is the proliferation of serv Troubleshooting always begins with information gathering. While much attention has been paid to centralizing machine data (e.g., logs, metrics), much less attention has been given to the human aspect of service discovery. Who owns a particular service? What Slack channel does the team work on? Where is the source for the service? What issues are currently known and being tracked? -## Kubernetes Annotations +## Kubernetes annotations Kubernetes annotations are designed to solve exactly this problem. Oft-overlooked, Kubernetes annotations are designed to add metadata to Kubernetes objects. The Kubernetes documentation says annotations can “attach arbitrary non-identifying metadata to objects.” This means that annotations should be used for attaching metadata that is external to Kubernetes (i.e., metadata that Kubernetes won’t use to identify objects. As such, annotations can contain any type of data. This is a contrast to labels, which are designed for uses internal to Kubernetes. As such, label structure and values are [constrained](/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) so they can be efficiently used by Kubernetes. @@ -87,9 +87,9 @@ Adopting a common convention for annotations ensures consistency and understanda ## Visualizing annotations: service catalogs -As the number of microservices and annotations proliferate, running `kubectl describe` can get tedious. Moreover, using `kubectl describe` requires every developer to have some direct access to the Kubernetes cluster. An [service catalog](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog), in the Ambassador meaning of that term, presents an internal, developer-oriented view of all services. +As the number of microservices and annotations proliferate, running `kubectl describe` can get tedious. Moreover, using `kubectl describe` requires every developer to have some direct access to the Kubernetes cluster. Over the past few years, [service catalogs](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog) have gained greater visibility in the Kubernetes ecosystem. Popularized by tools such as [Shopify's ServicesDB](https://shopify.engineering/scaling-mobile-development-by-treating-apps-as-services) and [Spotify's System Z](https://dzone.com/articles/modeling-microservices-at-spotify-with-petter-mari), service catalogs are internally-facing developer portals that present critical information about microservices. Note that these service catalogs should not be confused with the [Kubernetes Service Catalog project](https://svc-cat.io/). Built on the Open Service Broker API, the Kubernetes Service Catalog enables Kubernetes operators to plug in different services (e.g., databases) to their cluster. -Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include Backstage and Ambassador. [Backstage](https://backstage.io/) is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while [Ambassador](https://www.getambassador.io/products/service-catalog) provides a turnkey, cloud-hosted solution that can be set up quickly. +Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include [Backstage]((https://backstage.io/) and [Ambassador Service Catalog](https://www.getambassador.io/products/service-catalog). Backstage is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while Ambassador Service Catalog provides a turnkey, cloud-hosted solution that can be set up quickly. ## Annotate your services now and thank yourself later Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Don't wait until something is on fire in production to start wishing you had implemented better metrics and also documented how to get in touch with the part of your organization that looks after it. From a3986feedd5056efe5d58914346080c15023871d Mon Sep 17 00:00:00 2001 From: Kornel Bicskei <48224080+kornelbicskei@users.noreply.github.com> Date: Mon, 1 Mar 2021 21:05:35 -0800 Subject: [PATCH 015/218] Update documentation for AWS ELB Add more details on the mechanics of security group annotations as incorrect use could potentially cause an outage. --- .../en/docs/concepts/services-networking/service.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index b7a7edcd38..55d156060f 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -906,11 +906,18 @@ There are other annotations to manage Classic Elastic Load Balancers that are de # value. Defaults to 5, must be between 2 and 60 service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-53fae93f" - # A list of existing security groups to be added to ELB created. Unlike the annotation - # service.beta.kubernetes.io/aws-load-balancer-extra-security-groups, this replaces all other security groups previously assigned to the ELB. + # A list of existing security groups to be configured on the ELB created. Unlike the annotation + # service.beta.kubernetes.io/aws-load-balancer-extra-security-groups, this replaces all other security groups previously assigned to the ELB and also overrides the creation + # of a uniquely generated security group for this ELB. + # The first security group ID on this list is used as a source to permit incoming traffic to target worker nodes (service traffic and health checks). + # If multiple ELBs are configured with the same security group ID, only a single permit line will be added to the worker node security groups, that means if you delete any + # of those ELBs it will remove the single permit line and block access for all ELBs that shared the same security group ID. + # This can cause a cross-service outage if not used properly service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-53fae93f,sg-42efd82e" - # A list of additional security groups to be added to the ELB + # A list of additional security groups to be added to the created ELB, this leaves the uniquely generated security group in place, this ensures that every ELB + # has a unique security group ID and a matching permit line to allow traffic to the target worker nodes (service traffic and health checks). + # Security groups defined here can be shared between services. service.beta.kubernetes.io/aws-load-balancer-target-node-labels: "ingress-gw,gw-name=public-api" # A comma separated list of key-value pairs which are used From 2300a732101af99e171b669d5d8bde07b5c28465 Mon Sep 17 00:00:00 2001 From: Richard Li Date: Tue, 2 Mar 2021 09:01:39 -0500 Subject: [PATCH 016/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Tim Bannister --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index d2c4223713..922debf381 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -25,7 +25,7 @@ Kubernetes annotations are designed to solve exactly this problem. Oft-overlooke Here is an example. Imagine you have a Kubernetes service for quoting, called the quote service. You can do the following: ``` -$ kubectl annotate service quote a8r.io/owner=”@sally” +kubectl annotate service quote a8r.io/owner=”@sally” ``` In this example, we’ve just added an annotation called `a8r.io/owner` with the value of @sally. Now, we can use `kubectl describe` to get the information. From 48bd9a6a441db79c63a2080f6af7c50a2f15026e Mon Sep 17 00:00:00 2001 From: Richard Li Date: Tue, 2 Mar 2021 09:10:11 -0500 Subject: [PATCH 017/218] a few more fixes --- .../2021-02-27-annotating-k8s-for-humans.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index d2c4223713..81b14038e5 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -31,7 +31,6 @@ $ kubectl annotate service quote a8r.io/owner=”@sally” In this example, we’ve just added an annotation called `a8r.io/owner` with the value of @sally. Now, we can use `kubectl describe` to get the information. ``` -$ kubectl describe svc quote Name: quote Namespace: default Labels: @@ -48,7 +47,7 @@ Events: If you’re practicing GitOps (and you should be!) you’ll want to code these values directly into your Kubernetes manifest, e.g., -``` +```yaml apiVersion: v1 kind: Service metadata: @@ -68,20 +67,21 @@ spec: Adopting a common convention for annotations ensures consistency and understandability. Typically, you’ll want to attach the annotation to the service object, as services are the high-level resource that maps most clearly to a team’s responsibility. Namespacing your annotations is also very important. Here is one set of conventions, documented at [a8r.io](https://a8r.io), and reproduced below: +{{< table caption="Annotation convention for human-readable services">}} | Annotation | Description | | ------------------------------------------ | ------------------------------------------- | | `a8r.io/description` | Unstructured text description of the service for humans. | | `a8r.io/owner` | SSO username (GitHub), email address (linked to GitHub account), or unstructured owner description. | -| `a8r.io/chat` | Slack channel, or link to external chat system | -| `a8r.io/bugs` | Link to external bug tracker | -| `a8r.io/logs` | Link to external log viewer | -| `a8r.io/documentation` | Link to external project documentation | -| `a8r.io/repository` | Link to external VCS repository | -| `a8r.io/support` | Link to external support center | -| `a8r.io/runbook` | Link to external project runbook | -| `a8r.io/incidents` | Link to external incident dashboard | -| `a8r.io/uptime` | Link to external uptime dashboard | -| `a8r.io/performance` | Link to external performance dashboard | +| `a8r.io/chat` | Slack channel, or link to external chat system. | +| `a8r.io/bugs` | Link to external bug tracker. | +| `a8r.io/logs` | Link to external log viewer. | +| `a8r.io/documentation` | Link to external project documentation. | +| `a8r.io/repository` | Link to external VCS repository. | +| `a8r.io/support` | Link to external support center. | +| `a8r.io/runbook` | Link to external project runbook. | +| `a8r.io/incidents` | Link to external incident dashboard. | +| `a8r.io/uptime` | Link to external uptime dashboard. | +| `a8r.io/performance` | Link to external performance dashboard. | | `a8r.io/dependencies` | Unstructured text describing the service dependencies for humans. | From 5829a5d9584640b727b9420967a0189c6862e5b3 Mon Sep 17 00:00:00 2001 From: Richard Li Date: Thu, 4 Mar 2021 14:48:45 -0500 Subject: [PATCH 018/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md Co-authored-by: Kaitlyn Barnard --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index a9a65c9dae..b81c13a7f0 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -89,7 +89,7 @@ Adopting a common convention for annotations ensures consistency and understanda As the number of microservices and annotations proliferate, running `kubectl describe` can get tedious. Moreover, using `kubectl describe` requires every developer to have some direct access to the Kubernetes cluster. Over the past few years, [service catalogs](https://www.getambassador.io/learn/kubernetes-glossary/service-catalog) have gained greater visibility in the Kubernetes ecosystem. Popularized by tools such as [Shopify's ServicesDB](https://shopify.engineering/scaling-mobile-development-by-treating-apps-as-services) and [Spotify's System Z](https://dzone.com/articles/modeling-microservices-at-spotify-with-petter-mari), service catalogs are internally-facing developer portals that present critical information about microservices. Note that these service catalogs should not be confused with the [Kubernetes Service Catalog project](https://svc-cat.io/). Built on the Open Service Broker API, the Kubernetes Service Catalog enables Kubernetes operators to plug in different services (e.g., databases) to their cluster. -Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include [Backstage]((https://backstage.io/) and [Ambassador Service Catalog](https://www.getambassador.io/products/service-catalog). Backstage is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while Ambassador Service Catalog provides a turnkey, cloud-hosted solution that can be set up quickly. +Historically, service catalogs have not been a key piece of infrastructure, as applications consisted of a single monolith. Today, with distributed development and microservices the norm, service catalogs are increasingly being adopted as a best practice. Some popular options today include [Backstage](https://backstage.io/) and [Ambassador Service Catalog](https://www.getambassador.io/products/service-catalog). Backstage is a highly flexible project ideal for organizations that wish to customize every aspect of their service catalog, while Ambassador Service Catalog provides a turnkey, cloud-hosted solution that can be set up quickly. ## Annotate your services now and thank yourself later Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Don't wait until something is on fire in production to start wishing you had implemented better metrics and also documented how to get in touch with the part of your organization that looks after it. From ce4d342369f3c61ed5a73751f586c3cad26c8f55 Mon Sep 17 00:00:00 2001 From: Soichiro KAWAMURA Date: Sun, 7 Mar 2021 23:00:31 +0900 Subject: [PATCH 019/218] add Japanese translation of pod-priority-preemption.md --- .../configuration/pod-priority-preemption.md | 241 ++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 content/ja/docs/concepts/configuration/pod-priority-preemption.md diff --git a/content/ja/docs/concepts/configuration/pod-priority-preemption.md b/content/ja/docs/concepts/configuration/pod-priority-preemption.md new file mode 100644 index 0000000000..eb2ee65956 --- /dev/null +++ b/content/ja/docs/concepts/configuration/pod-priority-preemption.md @@ -0,0 +1,241 @@ +--- +title: Podの優先度とプリエンプション +content_type: concept +weight: 70 +--- + + + +{{< feature-state for_k8s_version="v1.14" state="stable" >}} + +[Pod](/docs/concepts/workloads/pods/)は _priority_（優先度）を持つことができます。 +優先度は他のPodに対する相対的なPodの重要度を示します。 +もしPodをスケジューリングできないときには、スケジューラーはそのPodをスケジューリングできるようにするため、優先度の低いPodをプリエンプトする（追い出す）ことを試みます。 + + + + + + +{{< warning >}} +クラスターの全てのユーザーが信用されていない場合、悪意のあるユーザーが可能な範囲で最も高い優先度のPodを作成することが可能です。これは他のPodが追い出されたりスケジューリングできない状態を招きます。 +管理者はResourceQuotaを使用して、ユーザーがPodを高い優先度で作成することを防ぐことができます。 + +詳細は[デフォルトで優先度クラスの消費を制限する](/docs/concepts/policy/resource-quotas/#limit-priority-class-consumption-by-default) +を参照してください。 +{{< /warning >}} + +## 優先度とプリエンプションを使う方法 + +優先度とプリエンプションを使うには、 + +1. 1つまたは複数の[PriorityClass](#priorityclass)を追加します + +1. 追加したPriorityClassを[`priorityClassName`](#pod-priority)に設定したPodを作成します。 + もちろんPodを直接作る必要はありません。 + 一般的には`priorityClassName`をDeploymentのようなコレクションオブジェクトのPodテンプレートに追加します。 + +これらの手順のより詳しい情報については、この先を読み進めてください。 + +{{< note >}} +Kubernetesには最初から既に2つのPriorityClassが設定された状態になっています。 +`system-cluster-critical`と`system-node-critical`です。 +これらは汎用のクラスであり、[重要なコンポーネントが常に最初にスケジュールされることを保証する](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/)ために使われます。 +{{< /note >}} + +## PriorityClass + +PriorityClassはnamespaceによらないオブジェクトで、優先度クラスの名称から優先度を表す整数値への対応を定義します。 +PriorityClassオブジェクトのメタデータの`name`フィールドにて名称を指定します。 +値は`value`フィールドで指定し、必須です。 +値が大きいほど、高い優先度を示します。 +PriorityClassオブジェクトの名称は[DNSサブドメイン名](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names)として適切であり、かつ`system-`から始まらないことが必要です。 + +PriorityClassオブジェクトは10億以下の任意の32ビットの整数値を持つことができます。 +それよりも大きな値は通常はプリエンプトや追い出すべきではない重要なシステム用のPodのために予約されています。 +クラスターの管理者は割り当てたい優先度に対して、PriorityClassオブジェクトを1つずつ作成すべきです。 + +PriorityClassは任意でフィールド`globalDefault`と`description`を設定可能です。 +`globalDefault`フィールドは`priorityClassName`が指定されないPodはこのPriorityClassを使うべきであることを示します。`globalDefault`が設定されたPriorityClassはシステムで一つのみ存在可能です。`globalDefault`が設定されたPriorityClassが存在しない場合は、`priorityClassName`が設定されていないPodの優先度は0に設定されます。 + +`description`フィールドは任意の文字列です。クラスターの利用者に対して、PriorityClassをどのような時に使うべきか示すことを意図しています。 + +### PodPriorityと既存のクラスターに関する注意 + +- もし既存のクラスターをこの機能がない状態でアップグレードすると、既存のPodの優先度は実質的に0になります。 + +- `globalDefault`が`true`に設定されたPriorityClassを追加しても、既存のPodの優先度は変わりません。PriorityClassのそのような値は、PriorityClassが追加された以後に作成されたPodのみに適用されます。 + +- PriorityClassを削除した場合、既存のPodは削除されたPriorityClassの名称を引き続き使用しますが、削除されたPriorityClassの名称を使うPodをそれ以上作成することはできなくなります。 + +### PriorityClassの例 + +```yaml +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: high-priority +value: 1000000 +globalDefault: false +description: "この優先度クラスはXYZサービスのPodに対してのみ使用すべきです。" +``` + +## 非プリエンプトのPriorityClass {#non-preempting-priority-class} + +{{< feature-state for_k8s_version="v1.19" state="beta" >}} + +`PreemptionPolicy: Never`と設定されたPodは、スケジューリングのキューにおいて他の優先度の低いPodよりも優先されますが、他のPodをプリエンプトすることはありません。 +スケジューリングされるのを待つ非プリエンプトのPodは、リソースが十分に利用可能になるまでスケジューリングキューに残ります。 +非プリエンプトのPodは、他のPodと同様に、スケジューラーのバックオフの対象になります。これは、スケジューラーがPodをスケジューリングしようと試みたものの失敗した場合、低い頻度で再試行するようにして、より優先度の低いPodが先にスケジューリングされることを許します。 + +非プリエンプトのPodは、他の優先度の高いPodにプリエンプトされる可能性はあります。 + +`PreemptionPolicy`はデフォルトでは`PreemptLowerPriority`に設定されており、これが設定されているPodは優先度の低いPodをプリエンプトすることを許容します。これは既存のデフォルトの挙動です。 +`PreemptionPolicy`を`Never`に設定すると、これが設定されたPodはプリエンプトを行わないようになります。 + +ユースケースの例として、データサイエンスの処理を挙げます。 +ユーザーは他の処理よりも優先度を高くしたいジョブを追加できますが、そのとき既存の実行中のPodの処理結果をプリエンプトによって捨てさせたくはありません。 +`PreemptionPolicy: Never`が設定された優先度の高いジョブは、他の既にキューイングされたPodよりも先に、クラスターのリソースが「自然に」開放されたときにスケジューリングされます。 + +### 非プリエンプトのPriorityClassの例 + +```yaml +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: high-priority-nonpreempting +value: 1000000 +preemptionPolicy: Never +globalDefault: false +description: "この優先度クラスは他のPodをプリエンプトさせません。" +``` + +## Podの優先度 + +一つ以上のPriorityClassがあれば、仕様にPriorityClassを指定したPodを作成することができるようになります。優先度のアドミッションコントローラーは`priorityClassName`フィールドを使用し、優先度の整数値を設定します。PriorityClassが見つからない場合、そのPodの作成は拒否されます。 + +下記のYAMLは上記の例で作成したPriorityClassを使用するPodの設定の例を示します。優先度のアドミッションコントローラーは仕様を確認し、このPodの優先度は1000000であると設定します。 + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: nginx + labels: + env: test +spec: + containers: + - name: nginx + image: nginx + imagePullPolicy: IfNotPresent + priorityClassName: high-priority +``` + +### スケジューリング順序におけるPodの優先度の効果 + +Podの優先度が有効な場合、スケジューラーは待機状態のPodをそれらの優先度順に並べ、スケジューリングキューにおいてより優先度の低いPodよりも前に来るようにします。その結果、その条件を満たしたときには優先度の高いPodは優先度の低いPodより早くスケジューリングされます。優先度の高いPodがスケジューリングできない場合は、スケジューラーは他の優先度の低いPodのスケジューリングも試みます。 + +## プリエンプション + +Podが作成されると、スケジューリング待ちのキューに入り待機状態になります。スケジューラーはキューからPodを取り出し、ノードへのスケジューリングを試みます。Podに指定された条件を全て満たすノードが見つからない場合は、待機状態のPodのためにプリエンプションロジックが発動します。待機状態のPodをPと呼ぶことにしましょう。プリエンプションロジックはPよりも優先度の低いPodを一つ以上追い出せばPをスケジューリングできるようになるノードを探します。そのようなノードがあれば、優先度の低いPodはノードから追い出されます。Podが追い出された後に、Pはノードへスケジューリング可能になります。 + +### ユーザーへ開示される情報 + +Pod PがノードNのPodをプリエンプトした場合、ノードNの名称がPのステータスの`nominatedNodeName`フィールドに設定されます。このフィールドはスケジューラーがPod Pのために予約しているリソースの追跡を助け、ユーザーにクラスターにおけるプリエンプトに関する情報を与えます。 + +Pod Pは「指名したノード」へスケジューリングされる必要はないことに注意してください。Podがプリエンプトされると、そのPodは猶予期間を得ます。スケジューラーがPodの終了を待つ間に他のノードが利用可能になると、スケジューラーは他のノードをPod Pのスケジューリング先にします。この結果、Podの`nominatedNodeName`と`nodeName`は必ずしも一致しません。また、スケジューラーがノードNのPodをプリエンプトさせた後に、Pod Pよりも優先度の高いPodが来た場合、スケジューラーはノードNをその新しい優先度の高いPodへ与えます。このような場合は、スケジューラーはPod Pの`nominatedNodeName`を消去します。これによって、スケジューラーはPod Pが他のノードのPodをプリエンプトさせられるようにします。 + +### プリエンプトの制限 + +#### プリエンプトされるPodの正常終了 + +Podがプリエンプトされると、[猶予期間](/ja/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)が与えられます。 + +Podは作業を完了し、終了するために十分な時間が与えられます。仮にそうでない場合、強制終了されます。この猶予期間によって、スケジューラーがPodをプリエンプトした時刻と、待機状態のPod Pがノード Nにスケジュール可能になるまでの時刻の間に間が開きます。この間、スケジューラーは他の待機状態のPodをスケジュールしようと試みます。プリエンプトされたPodが終了したら、スケジューラーは待ち行列にあるPodをスケジューリングしようと試みます。そのため、Podがプリエンプトされる時刻と、Pがスケジュールされた時刻には間が開くことが一般的です。この間を最小にするには、優先度の低いPodの猶予期間を0または小さい値にする方法があります。 + +#### PodDisruptionBudgetは対応するが、保証されない + +[PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) (PDB)は、アプリケーションのオーナーが冗長化されたアプリケーションのPodが意図的に中断される数の上限を設定できるようにするものです。KubernetesはPodをプリエンプトする際にPDBに対応しますが、PDBはベストエフォートで考慮します。スケジューラーはプリエンプトさせたとしてもPDBに違反しないPodを探します。そのようなPodが見つからない場合でもプリエンプションは実行され、PDBに反しますが優先度の低いPodが追い出されます。 + +#### 優先度の低いPodにおけるPod間のアフィニティ + +次の条件が真の場合のみ、ノードはプリエンプションの候補に入ります。 +「待機状態のPodよりも優先度の低いPodをノードから全て追い出したら、待機状態のPodをノードへスケジュールできるか」 + +{{< note >}} +プリエンプションは必ずしも優先度の低いPodを全て追い出しません。 +優先度の低いPodを全て追い出さなくても待機状態のPodがスケジューリングできる場合、一部のPodのみ追い出されます。 +このような場合であったとしても、上記の条件は真である必要があります。偽であれば、そのノードはプリエンプションの対象とはされません。 +{{< /note >}} + +待機状態のPodが、優先度の低いPodとの間でPod間のアフィニティを持つ場合、Pod間のアフィニティはそれらの優先度の低いPodがなければ満たされません。この場合、スケジューラーはノードのどのPodもプリエンプトしようとはせず、代わりに他のノードを探します。スケジューラーは適切なノードを探せる場合と探せない場合があります。この場合、待機状態のPodがスケジューリングされる保証はありません。 + +この問題に対して推奨される解決策は、優先度が同一または高いPodに対してのみPod間のアフィニティを作成することです。 + +#### 複数ノードに対するプリエンプション + +Pod PがノードNにスケジューリングできるよう、ノードNがプリエンプションの対象となったとします。 +他のノードのPodがプリエンプトされた場合のみPが実行可能になることもあります。下記に例を示します。 + +* Pod PをノードNに配置することを検討します。 +* Pod QはノードNと同じゾーンにある別のノードで実行中です。 +* Pod Pはゾーンに対するQへのアンチアフィニティを持ちます (`topologyKey: topology.kubernetes.io/zone`)。 +* Pod Pと、ゾーン内の他のPodに対しては他のアンチアフィニティはない状態です。 +* Pod PをノードNへスケジューリングするには、Pod Qをプリエンプトすることが考えられますが、スケジューラーは複数ノードにわたるプリエンプションは行いません。そのため、Pod PはノードNへはスケジューリングできないとみなされます。 + +Pod Qがそのノードから追い出されると、Podアンチアフィニティに違反しなくなるので、Pod PはノードNへスケジューリング可能になります。 + +複数ノードに対するプリエンプションに関しては、十分な需要があり、合理的な性能を持つアルゴリズムを見つけられた場合に、追加することを検討する可能性があります。 + +## トラブルシューティング + +Podの優先度とプリエンプションは望まない副作用をもたらす可能性があります。 +いくつかの起こりうる問題と、その対策について示します。 + +### Podが不必要にプリエンプトされる + +プリエンプションは、リソースが不足している場合に優先度の高い待機状態のPodのためにクラスターの既存のPodを追い出します。 +誤って高い優先度をPodに割り当てると、意図しない高い優先度のPodはクラスター内でプリエンプションを引き起こす可能性があります。Podの優先度はPodの仕様の`priorityClassName`フィールドにて指定されます。優先度を示す整数値へと変換された後、`podSpec`の`priority`へ設定されます。 + +この問題に対処するには、Podの`priorityClassName`をより低い優先度に変更するか、このフィールドを未設定にすることができます。`priorityClassName`が未設定の場合、デフォルトでは優先度は0とされます。 + +Podがプリエンプトされたとき、プリエンプトされたPodのイベントが記録されます。 +プリエンプションはPodに必要なリソースがクラスターにない場合のみ起こるべきです。 +このような場合、プリエンプションはプリエンプトされるPodよりも待機状態のPodの優先度が高い場合のみ発生します。 +プリエンプションは待機状態のPodがない場合や待機状態のPodがプリエンプト対象のPod以下の優先度を持つ場合には決して発生しません。そのような状況でプリエンプションが発生した場合、問題を報告してください。 + +### Podはプリエンプトされたが、プリエンプトさせたPodがスケジューリングされない + +Podがプリエンプトされると、それらのPodが要求した猶予期間が与えられます。そのデフォルトは30秒です。 +Podがその期間内に終了しない場合、強制終了されます。プリエンプトされたPodがなくなれば、プリエンプトさせたPodはスケジューリング可能です。 + +プリエンプトさせたPodがプリエンプトされたPodの終了を待っている間に、より優先度の高いPodが同じノードに対して作成されることもあります。この場合、スケジューラーはプリエンプトさせたPodの代わりに優先度の高いPodをスケジューリングします。 + +これは予期された挙動です。優先度の高いPodは優先度の低いPodに取って代わります。 + +### 優先度の高いPodが優先度の低いPodより先にプリエンプトされる + +スケジューラーは待機状態のPodが実行可能なノードを探します。ノードが見つからない場合、スケジューラーは任意のノードから優先度の低いPodを追い出し、待機状態のPodのためのリソースを確保しようとします。 +仮に優先度の低いPodが動いているノードが待機状態のPodを動かすために適切ではない場合、スケジューラーは他のノードで動いているPodと比べると、優先度の高いPodが動いているノードをプリエンプションの対象に選ぶことがあります。この場合もプリエンプトされるPodはプリエンプトを起こしたPodよりも優先度が低い必要があります。 + +複数のノードがプリエンプションの対象にできる場合、スケジューラーは優先度が最も低いPodのあるノードを選ぼうとします。しかし、そのようなPodがPodDisruptionBudgetを持っており、プリエンプトするとPDBに反する場合はスケジューラーは優先度の高いPodのあるノードを選ぶこともあります。 + +複数のノードがプリエンプションの対象として利用可能で、上記の状況に当てはまらない場合、スケジューラーは優先度の最も低いノードを選択します。 + +## Podの優先度とQoSの相互作用 {#interactions-of-pod-priority-and-qos} + +Podの優先度と{{< glossary_tooltip text="QoSクラス" term_id="qos-class" >}}は直交する機能で、わずかに相互作用がありますが、デフォルトではQoSクラスによる優先度の設定の制約はありません。スケジューラーのプリエンプションのロジックはプリエンプションの対象を決めるときにQoSクラスは考慮しません。 +プリエンプションはPodの優先度を考慮し、優先度が最も低いものを候補とします。より優先度の高いPodは優先度の低いPodを追い出すだけではプリエンプトを起こしたPodのスケジューリングに不十分な場合と、`PodDisruptionBudget`により優先度の低いPodが保護されている場合のみ対象になります。 + +QoSとPodの優先度の両方を考慮するコンポーネントは[リソース不足によりkubeletがPodを追い出す](/docs/tasks/administer-cluster/out-of-resource/)のみです。 +kubeletは追い出すPodの順位付けを次の順で行います。枯渇したリソースを要求以上に使用しているか、優先度、枯渇したリソースの消費量の複数のPodの要求に対する相対値。 +詳細は[エンドユーザーのPodの追い出し](/docs/tasks/administer-cluster/out-of-resource/#evicting-end-user-pods)を参照してください。 + + +kubeletによるリソース不足時のPodの追い出しでは、リソースの消費が要求を超えないPodは追い出されません。優先度の低いPodのリソースの利用量がその要求を超えていなければ、追い出されることはありません。より優先度が高く、要求を超えてリソースを使用しているPodが追い出されます。 + + +## {{% heading "whatsnext" %}} + +* PriorityClassと関連付けてResourceQuotaを使用することに関して [デフォルトで優先度クラスの消費を制限する](/ja/docs/concepts/policy/resource-quotas/#limit-priority-class-consumption-by-default) + From cf19c5d74b5d86d0630915c5d927fe5329c065be Mon Sep 17 00:00:00 2001 From: CKchen0726 Date: Tue, 9 Mar 2021 12:30:35 +0800 Subject: [PATCH 020/218] modify some links in Kustomize Feature List --- .../tasks/manage-kubernetes-objects/kustomization.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md b/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md index 7c59052ffa..eb6d638603 100644 --- a/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md +++ b/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md @@ -815,14 +815,14 @@ deployment.apps "dev-my-nginx" deleted | commonLabels | map[string]string | labels to add to all resources and selectors | | commonAnnotations | map[string]string | annotations to add to all resources | | resources | []string | each entry in this list must resolve to an existing resource configuration file | -| configmapGenerator | [][ConfigMapArgs](https://github.com/kubernetes-sigs/kustomize/blob/release-kustomize-v4.0/api/types/kustomization.go#L99) | Each entry in this list generates a ConfigMap | -| secretGenerator | [][SecretArgs](https://github.com/kubernetes-sigs/kustomize/blob/release-kustomize-v4.0/api/types/kustomization.go#L106) | Each entry in this list generates a Secret | -| generatorOptions | [GeneratorOptions](https://github.com/kubernetes-sigs/kustomize/blob/release-kustomize-v4.0/api/types/kustomization.go#L109) | Modify behaviors of all ConfigMap and Secret generator | +| configMapGenerator | [][ConfigMapArgs](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/configmapargs.go#L7) | Each entry in this list generates a ConfigMap | +| secretGenerator | [][SecretArgs](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/secretargs.go#L7) | Each entry in this list generates a Secret | +| generatorOptions | [GeneratorOptions](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/generatoroptions.go#L7) | Modify behaviors of all ConfigMap and Secret generator | | bases | []string | Each entry in this list should resolve to a directory containing a kustomization.yaml file | | patchesStrategicMerge | []string | Each entry in this list should resolve a strategic merge patch of a Kubernetes object | -| patchesJson6902 | [][Json6902](https://github.com/kubernetes-sigs/kustomize/blob/release-kustomize-v4.0/api/types/patchjson6902.go#L8) | Each entry in this list should resolve to a Kubernetes object and a Json Patch | -| vars | [][Var](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/var.go#L31) | Each entry is to capture text from one resource's field | -| images | [][Image](https://github.com/kubernetes-sigs/kustomize/tree/master/api/types/image.go#L23) | Each entry is to modify the name, tags and/or digest for one image without creating patches | +| patchesJson6902 | [][Patch](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/patch.go#L10) | Each entry in this list should resolve to a Kubernetes object and a Json Patch | +| vars | [][Var](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/var.go#L19) | Each entry is to capture text from one resource's field | +| images | [][Image](https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/image.go#L8) | Each entry is to modify the name, tags and/or digest for one image without creating patches | | configurations | []string | Each entry in this list should resolve to a file containing [Kustomize transformer configurations](https://github.com/kubernetes-sigs/kustomize/tree/master/examples/transformerconfigs) | | crds | []string | Each entry in this list should resolve to an OpenAPI definition file for Kubernetes types | From ec5b641a38ac304b24ba1d4ddc8ce66450a4bf24 Mon Sep 17 00:00:00 2001 From: Naoki Oketani Date: Mon, 15 Mar 2021 20:54:50 +0900 Subject: [PATCH 021/218] Translate docs/tasks/configmap-secret/managing-secret-using-config-file/ into Japanese --- .../managing-secret-using-config-file.md | 183 ++++++++++++++++++ .../managing-secret-using-kubectl.md | 2 +- 2 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md diff --git a/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md b/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md new file mode 100644 index 0000000000..170fa6bd9d --- /dev/null +++ b/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md @@ -0,0 +1,183 @@ +--- +title: 設定ファイルを使用してSecretを管理する +content_type: task +weight: 20 +description: リソース設定ファイルを使用してSecretを作成する +--- + + + +## {{% heading "prerequisites" %}} + +{{< include "task-tutorial-prereqs.md" >}} + + + +## 設定ファイルを作成する + +先にJSONやYAML形式のファイルでSecretを作成してから、そのオブジェクトを作成することができます。 +[Secret](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#secret-v1-core)リソースには、`data`と`stringData`の2つのマップが含まれています。 +`data`フィールドは任意のデータを格納するのに使用され、base64でエンコードされます。 +`stringData`フィールドは利便性のために用意されており、Secretデータをエンコードされていない文字列として提供することができます。 +`data`と`stringData`のキーは、英数字、`-`、`_`、`.`で構成されている必要があります。 + +たとえば、`data`フィールドを使用して2つの文字列をSecretに格納するには、次のように文字列をbase64に変換します: + +```shell +echo -n 'admin' | base64 +``` + +出力は次のようになります: + +``` +YWRtaW4= +``` + +```shell +echo -n '1f2d1e2e67df' | base64 +``` + +出力は次のようになります: + +``` +MWYyZDFlMmU2N2Rm +``` + +以下のようなSecret設定ファイルを記述します: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mysecret +type: Opaque +data: + username: YWRtaW4= + password: MWYyZDFlMmU2N2Rm +``` + +なお、Secretオブジェクトの名前は、有効な[DNSサブドメイン名](/ja/docs/concepts/overview/working-with-objects/names#dns-subdomain-names)である必要があります。 + +{{< note >}} +SecretデータのシリアライズされたJSONおよびYAMLの値は、base64文字列としてエンコードされます。 +これらの文字列の中では、改行は有効ではなく、省略する必要があります。 +Darwin/macOSで`base64`ユーティリティーを使用する場合、長い行を分割するために`-b`オプションを使用するのは避けるべきです。 +逆に、Linux ユーザーは、`base64` コマンドにオプション`-w 0`を追加するか、`-w`オプションが利用できない場合には、パイプライン`base64 | tr -d '\n'`を追加する*必要があります*。 +{{< /note >}} + +特定のシナリオでは、代わりに`stringData`フィールドを使用できます。 +このフィールドでは、base64エンコードされていない文字列を直接Secretに入れることができ、Secretの作成時や更新時には、その文字列がエンコードされます。 + +たとえば、設定ファイルを保存するためにSecretを使用しているアプリケーションをデプロイする際に、デプロイプロセス中に設定ファイルの一部を入力したい場合などが考えられます。 + +たとえば、次のような設定ファイルを使用しているアプリケーションの場合: + +```yaml +apiUrl: "https://my.api.com/api/v1" +username: "" +password: "" +``` + +次のような定義でSecretに格納できます: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mysecret +type: Opaque +stringData: + config.yaml: | + apiUrl: "https://my.api.com/api/v1" + username: + password: +``` + +## Secretを作成する + +[`kubectl apply`](/docs/reference/generated/kubectl/kubectl-commands#apply)でSecretを作成します: + +```shell +kubectl apply -f ./secret.yaml +``` + +出力は次のようになります: + +``` +secret/mysecret created +``` + +## Secretを確認する + +`stringData`フィールドは、書き込み専用の便利なフィールドです。Secretを取得する際には決して出力されません。たとえば、次のようなコマンドを実行した場合: + +```shell +kubectl get secret mysecret -o yaml +``` + +出力は次のようになります: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + creationTimestamp: 2018-11-15T20:40:59Z + name: mysecret + namespace: default + resourceVersion: "7225" + uid: c280ad2e-e916-11e8-98f2-025000000001 +type: Opaque +data: + config.yaml: YXBpVXJsOiAiaHR0cHM6Ly9teS5hcGkuY29tL2FwaS92MSIKdXNlcm5hbWU6IHt7dXNlcm5hbWV9fQpwYXNzd29yZDoge3twYXNzd29yZH19 +``` + +`kubectl get`と`kubectl describe`コマンドはデフォルトではSecretの内容を表示しません。 +これは、Secretが不用意に他人にさらされたり、ターミナルログに保存されたりしないようにするためです。 +エンコードされたデータの実際の内容を確認するには、[Secretのデコード](/ja/docs/tasks/configmap-secret/managing-secret-using-kubectl/#decoding-secret)を参照してください。 + +`username`などのフィールドが`data`と`stringData`の両方に指定されている場合は、`stringData`の値が使われます。 +たとえば、以下のようなSecretの定義の場合: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mysecret +type: Opaque +data: + username: YWRtaW4= +stringData: + username: administrator +``` + +結果は以下の通りです: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + creationTimestamp: 2018-11-15T20:46:46Z + name: mysecret + namespace: default + resourceVersion: "7579" + uid: 91460ecb-e917-11e8-98f2-025000000001 +type: Opaque +data: + username: YWRtaW5pc3RyYXRvcg== +``` + +`YWRtaW5pc3RyYXRvcg==`をデコードすると`administrator`となります。 + +## クリーンアップ + +作成したSecretを削除するには次のコマンドを実行します: + +```shell +kubectl delete secret mysecret +``` + +## {{% heading "whatsnext" %}} + +- [Secretのコンセプト](/ja/docs/concepts/configuration/secret/)を読む +- [kubectlを使用してSecretを管理する](/ja/docs/tasks/configmap-secret/managing-secret-using-kubectl/)方法を知る +- [kustomizeを使用してSecretを管理する](/docs/tasks/configmap-secret/managing-secret-using-kustomize/)方法を知る diff --git a/content/ja/docs/tasks/configmap-secret/managing-secret-using-kubectl.md b/content/ja/docs/tasks/configmap-secret/managing-secret-using-kubectl.md index fb8c89c1e3..9e498de8ac 100644 --- a/content/ja/docs/tasks/configmap-secret/managing-secret-using-kubectl.md +++ b/content/ja/docs/tasks/configmap-secret/managing-secret-using-kubectl.md @@ -142,5 +142,5 @@ kubectl delete secret db-user-pass ## {{% heading "whatsnext" %}} - [Secretのコンセプト](/ja/docs/concepts/configuration/secret/)を読む -- [設定ファイルを使用してSecretを管理する](/docs/tasks/configmap-secret/managing-secret-using-config-file/)方法を知る +- [設定ファイルを使用してSecretを管理する](/ja/docs/tasks/configmap-secret/managing-secret-using-config-file/)方法を知る - [kustomizeを使用してSecretを管理する](/docs/tasks/configmap-secret/managing-secret-using-kustomize/)方法を知る From 4dcbe54fb6e6034493d43bb3e2726c56f92be9c4 Mon Sep 17 00:00:00 2001 From: Naoki Oketani Date: Wed, 17 Mar 2021 20:26:20 +0900 Subject: [PATCH 022/218] apply suggestion --- .../configmap-secret/managing-secret-using-config-file.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md b/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md index 170fa6bd9d..fad8ab80ac 100644 --- a/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md +++ b/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md @@ -15,7 +15,7 @@ description: リソース設定ファイルを使用してSecretを作成する ## 設定ファイルを作成する -先にJSONやYAML形式のファイルでSecretを作成してから、そのオブジェクトを作成することができます。 +あらかじめYAMLまたはJSON形式でSecretのマニフェストを作成したうえで、オブジェクトを作成することができます。 [Secret](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#secret-v1-core)リソースには、`data`と`stringData`の2つのマップが含まれています。 `data`フィールドは任意のデータを格納するのに使用され、base64でエンコードされます。 `stringData`フィールドは利便性のために用意されており、Secretデータをエンコードされていない文字列として提供することができます。 @@ -60,7 +60,7 @@ data: {{< note >}} SecretデータのシリアライズされたJSONおよびYAMLの値は、base64文字列としてエンコードされます。 -これらの文字列の中では、改行は有効ではなく、省略する必要があります。 +文字列の中の改行は不正で、含まれていてはなりません。 Darwin/macOSで`base64`ユーティリティーを使用する場合、長い行を分割するために`-b`オプションを使用するのは避けるべきです。 逆に、Linux ユーザーは、`base64` コマンドにオプション`-w 0`を追加するか、`-w`オプションが利用できない場合には、パイプライン`base64 | tr -d '\n'`を追加する*必要があります*。 {{< /note >}} From 04c41cb89fe0f3460a07035675a032dab567d545 Mon Sep 17 00:00:00 2001 From: Naoki Oketani Date: Wed, 17 Mar 2021 20:27:58 +0900 Subject: [PATCH 023/218] apply suggestion --- .../tasks/configmap-secret/managing-secret-using-config-file.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md b/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md index fad8ab80ac..f9572ca1f4 100644 --- a/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md +++ b/content/ja/docs/tasks/configmap-secret/managing-secret-using-config-file.md @@ -60,7 +60,7 @@ data: {{< note >}} SecretデータのシリアライズされたJSONおよびYAMLの値は、base64文字列としてエンコードされます。 -文字列の中の改行は不正で、含まれていてはなりません。 +文字列中の改行は不正で、含まれていてはなりません。 Darwin/macOSで`base64`ユーティリティーを使用する場合、長い行を分割するために`-b`オプションを使用するのは避けるべきです。 逆に、Linux ユーザーは、`base64` コマンドにオプション`-w 0`を追加するか、`-w`オプションが利用できない場合には、パイプライン`base64 | tr -d '\n'`を追加する*必要があります*。 {{< /note >}} From 62e83f84a02a14512e02ca2f7464576edd9e6d70 Mon Sep 17 00:00:00 2001 From: Ryuichi KAWAMATA Date: Sun, 21 Mar 2021 22:39:06 +0900 Subject: [PATCH 024/218] Update ja/docs/concepts/containers/_index.md --- content/ja/docs/concepts/containers/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/concepts/containers/_index.md b/content/ja/docs/concepts/containers/_index.md index 5b10416c0f..fd3506ea40 100755 --- a/content/ja/docs/concepts/containers/_index.md +++ b/content/ja/docs/concepts/containers/_index.md @@ -21,7 +21,7 @@ no_list: true ## コンテナイメージ [コンテナイメージ](/docs/concepts/containers/images/)はすぐに実行可能なソフトウェアパッケージで、アプリケーションの実行に必要なものをすべて含んています。コードと必要なランタイム、アプリケーションとシステムのライブラリ、そして必須な設定項目のデフォルト値を含みます。 -設計上、コンテナは不変で、既に実行中のコンテナのコードを変更することはできません。コンテナ化されたアプリケーションがあり変更したい場合は、変更を含んだ新しいコンテナをビルドし、コンテナを再作成して、更新されたイメージから起動する必要があります。 +設計上、コンテナは不変で、既に実行中のコンテナのコードを変更することはできません。コンテナ化されたアプリケーションがあり変更したい場合は、変更を含んだ新しいイメージをビルドし、コンテナを再作成して、更新されたイメージから起動する必要があります。 ## コンテナランタイム From 81be1fe0f6e18713d45fb0e8149a9be86b00c3d7 Mon Sep 17 00:00:00 2001 From: s-kawamura-w664 Date: Mon, 22 Mar 2021 02:58:36 +0000 Subject: [PATCH 025/218] delete a illegal code in fr. --- content/fr/docs/concepts/cluster-administration/logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/fr/docs/concepts/cluster-administration/logging.md b/content/fr/docs/concepts/cluster-administration/logging.md index 465ec742b1..0932de7406 100644 --- a/content/fr/docs/concepts/cluster-administration/logging.md +++ b/content/fr/docs/concepts/cluster-administration/logging.md @@ -82,7 +82,7 @@ conteneur a crashé. Si le Pod a plusieurs conteneurs, il faut spécifier le nom du conteneur dont on veut récupérer le journal d'évènement. Dans notre exemple le conteneur s'appelle -`count` donc vous pouvez utiliser `kubectl logs counter count`. Plus de détails +`count` donc vous pouvez utiliser `kubectl logs counter count`. Plus de détails dans la [documentation de `kubectl logs`] (/docs/reference/generated/kubectl/kubectl-commands#logs) From 4538d24527d31d9a28a3202c4e6567826d874df2 Mon Sep 17 00:00:00 2001 From: translucens Date: Thu, 25 Mar 2021 22:08:45 +0900 Subject: [PATCH 026/218] Apply suggestions from code review Co-authored-by: Kei Ak --- .../concepts/configuration/pod-priority-preemption.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/ja/docs/concepts/configuration/pod-priority-preemption.md b/content/ja/docs/concepts/configuration/pod-priority-preemption.md index eb2ee65956..b83d1c6867 100644 --- a/content/ja/docs/concepts/configuration/pod-priority-preemption.md +++ b/content/ja/docs/concepts/configuration/pod-priority-preemption.md @@ -56,7 +56,7 @@ PriorityClassオブジェクトは10億以下の任意の32ビットの整数値 クラスターの管理者は割り当てたい優先度に対して、PriorityClassオブジェクトを1つずつ作成すべきです。 PriorityClassは任意でフィールド`globalDefault`と`description`を設定可能です。 -`globalDefault`フィールドは`priorityClassName`が指定されないPodはこのPriorityClassを使うべきであることを示します。`globalDefault`が設定されたPriorityClassはシステムで一つのみ存在可能です。`globalDefault`が設定されたPriorityClassが存在しない場合は、`priorityClassName`が設定されていないPodの優先度は0に設定されます。 +`globalDefault`フィールドは`priorityClassName`が指定されないPodはこのPriorityClassを使うべきであることを示します。`globalDefault`がtrueに設定されたPriorityClassはシステムで一つのみ存在可能です。`globalDefault`が設定されたPriorityClassが存在しない場合は、`priorityClassName`が設定されていないPodの優先度は0に設定されます。 `description`フィールドは任意の文字列です。クラスターの利用者に対して、PriorityClassをどのような時に使うべきか示すことを意図しています。 @@ -66,7 +66,7 @@ PriorityClassは任意でフィールド`globalDefault`と`description`を設定 - `globalDefault`が`true`に設定されたPriorityClassを追加しても、既存のPodの優先度は変わりません。PriorityClassのそのような値は、PriorityClassが追加された以後に作成されたPodのみに適用されます。 -- PriorityClassを削除した場合、既存のPodは削除されたPriorityClassの名称を引き続き使用しますが、削除されたPriorityClassの名称を使うPodをそれ以上作成することはできなくなります。 +- PriorityClassを削除した場合、削除されたPriorityClassの名前を使用する既存のPodは変更されませんが、削除されたPriorityClassの名前を使うPodをそれ以上作成することはできなくなります。 ### PriorityClassの例 @@ -94,7 +94,7 @@ description: "この優先度クラスはXYZサービスのPodに対してのみ `PreemptionPolicy`を`Never`に設定すると、これが設定されたPodはプリエンプトを行わないようになります。 ユースケースの例として、データサイエンスの処理を挙げます。 -ユーザーは他の処理よりも優先度を高くしたいジョブを追加できますが、そのとき既存の実行中のPodの処理結果をプリエンプトによって捨てさせたくはありません。 +ユーザーは他の処理よりも優先度を高くしたいジョブを追加できますが、そのとき既存の実行中のPodの処理結果をプリエンプトによって破棄させたくはありません。 `PreemptionPolicy: Never`が設定された優先度の高いジョブは、他の既にキューイングされたPodよりも先に、クラスターのリソースが「自然に」開放されたときにスケジューリングされます。 ### 非プリエンプトのPriorityClassの例 @@ -143,7 +143,7 @@ Podが作成されると、スケジューリング待ちのキューに入り Pod PがノードNのPodをプリエンプトした場合、ノードNの名称がPのステータスの`nominatedNodeName`フィールドに設定されます。このフィールドはスケジューラーがPod Pのために予約しているリソースの追跡を助け、ユーザーにクラスターにおけるプリエンプトに関する情報を与えます。 -Pod Pは「指名したノード」へスケジューリングされる必要はないことに注意してください。Podがプリエンプトされると、そのPodは猶予期間を得ます。スケジューラーがPodの終了を待つ間に他のノードが利用可能になると、スケジューラーは他のノードをPod Pのスケジューリング先にします。この結果、Podの`nominatedNodeName`と`nodeName`は必ずしも一致しません。また、スケジューラーがノードNのPodをプリエンプトさせた後に、Pod Pよりも優先度の高いPodが来た場合、スケジューラーはノードNをその新しい優先度の高いPodへ与えます。このような場合は、スケジューラーはPod Pの`nominatedNodeName`を消去します。これによって、スケジューラーはPod Pが他のノードのPodをプリエンプトさせられるようにします。 +Pod Pは必ずしも「指名したノード」へスケジューリングされないことに注意してください。Podがプリエンプトされると、そのPodは終了までの猶予期間を得ます。スケジューラーがPodの終了を待つ間に他のノードが利用可能になると、スケジューラーは他のノードをPod Pのスケジューリング先にします。この結果、Podの`nominatedNodeName`と`nodeName`は必ずしも一致しません。また、スケジューラーがノードNのPodをプリエンプトさせた後に、Pod Pよりも優先度の高いPodが来た場合、スケジューラーはノードNをその新しい優先度の高いPodへ与えます。このような場合は、スケジューラーはPod Pの`nominatedNodeName`を消去します。これによって、スケジューラーはPod Pが他のノードのPodをプリエンプトさせられるようにします。 ### プリエンプトの制限 @@ -238,4 +238,3 @@ kubeletによるリソース不足時のPodの追い出しでは、リソース ## {{% heading "whatsnext" %}} * PriorityClassと関連付けてResourceQuotaを使用することに関して [デフォルトで優先度クラスの消費を制限する](/ja/docs/concepts/policy/resource-quotas/#limit-priority-class-consumption-by-default) - From 88e88a4c83aa004b33107bae4584c4a9a9d01f1c Mon Sep 17 00:00:00 2001 From: Ryo Nakao Date: Fri, 26 Mar 2021 10:59:18 +0900 Subject: [PATCH 027/218] Fix typo in Japanese doc --- content/ja/docs/reference/access-authn-authz/rbac.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/reference/access-authn-authz/rbac.md b/content/ja/docs/reference/access-authn-authz/rbac.md index fee25f323e..04409b3b8e 100644 --- a/content/ja/docs/reference/access-authn-authz/rbac.md +++ b/content/ja/docs/reference/access-authn-authz/rbac.md @@ -43,7 +43,7 @@ ClusterRolesにはいくつかの用途があります。ClusterRoleを利用し 2. Namespaceに属するリソースに対する権限を定義し、すべてのNamespaceにわたって付与する 3. クラスター単位でスコープされているリソースに対するアクセス許可を定義する -NamespaceでRoleを定義する場合は、Roleを使用します。クラスター全体でRoleを定義する婆は、ClusterRoleを使用します +NamespaceでRoleを定義する場合は、Roleを使用します。クラスター全体でRoleを定義する場合は、ClusterRoleを使用します #### Roleの例 From 50b14656e23a3e689b5fadaefd79c3e49cd9b901 Mon Sep 17 00:00:00 2001 From: Yuiko Mouri Date: Wed, 17 Mar 2021 10:35:02 +0900 Subject: [PATCH 028/218] Replace redirect links and fix link to use localized page of reference/index --- content/ja/docs/reference/_index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/ja/docs/reference/_index.md b/content/ja/docs/reference/_index.md index 0496a730c4..aca4c278b5 100644 --- a/content/ja/docs/reference/_index.md +++ b/content/ja/docs/reference/_index.md @@ -16,7 +16,7 @@ content_type: concept ## APIリファレンス -* [Kubernetes API概要](/docs/reference/using-api/api-overview/) - Kubernetes APIの概要です。 +* [Kubernetes API概要](/docs/reference/using-api/) - Kubernetes APIの概要です。 * [Kubernetes APIリファレンス {{< latest-version >}}](/docs/reference/generated/kubernetes-api/{{< latest-version >}}/) ## APIクライアントライブラリー @@ -30,9 +30,9 @@ content_type: concept ## CLIリファレンス -* [kubectl](/docs/reference/kubectl/overview/) - コマンドの実行やKubernetesクラスターの管理に使う主要なCLIツールです。 +* [kubectl](/ja/docs/reference/kubectl/overview/) - コマンドの実行やKubernetesクラスターの管理に使う主要なCLIツールです。 * [JSONPath](/ja/docs/reference/kubectl/jsonpath/) - kubectlで[JSONPath記法](https://goessner.net/articles/JsonPath/)を使うための構文ガイドです。 -* [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/) - セキュアなKubernetesクラスターを簡単にプロビジョニングするためのCLIツールです。 +* [kubeadm](ja/docs/reference/setup-tools/kubeadm/) - セキュアなKubernetesクラスターを簡単にプロビジョニングするためのCLIツールです。 ## コンポーネントリファレンス From 31dcd91c9f7e3b60142a4bec849af9bbbada9857 Mon Sep 17 00:00:00 2001 From: Jordan Christiansen Date: Thu, 25 Mar 2021 12:09:25 -0500 Subject: [PATCH 029/218] Document using configMapGenerators in Deployments --- .../kustomization.md | 139 ++++++++++++++++++ 1 file changed, 139 insertions(+) diff --git a/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md b/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md index 3ea3c50e8d..c28e5123ec 100644 --- a/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md +++ b/content/en/docs/tasks/manage-kubernetes-objects/kustomization.md @@ -114,6 +114,98 @@ metadata: name: example-configmap-2-g2hdhfc6tk ``` +To use a generated ConfigMap in a Deployment, reference it by the name of the configMapGenerator. Kustomize will automatically replace this name with the generated name. + +This is an example deployment that uses a generated ConfigMap: + +```yaml +# Create a application.properties file +cat <application.properties +FOO=Bar +EOF + +cat <deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + labels: + app: my-app +spec: + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: app + image: my-app + volumeMount: + - name: config + mountPath: /config + volumes: + - name: config + configMap: + name: example-configmap-1 +EOF + +cat <./kustomization.yaml +resources: +- deployment.yaml +configMapGenerator: +- name: example-configmap-1 + files: + - application.properties +EOF +``` + +Generate the ConfigMap and Deployment: + +```shell +kubectl kustomize ./ +``` + +The generated Deployment will refer to the generated ConfigMap by name: + +```yaml +apiVersion: v1 +data: + application.properties: | + FOO=Bar +kind: ConfigMap +metadata: + name: example-configmap-1-g4hk9g2ff8 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: my-app + name: my-app +spec: + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - image: my-app + name: app + volumeMount: + - mountPath: /config + name: config + volumes: + - configMap: + name: example-configmap-1-g4hk9g2ff8 + name: config +``` + #### secretGenerator You can generate Secrets from files or literal key-value pairs. To generate a Secret from a file, add an entry to the `files` list in `secretGenerator`. Here is an example of generating a Secret with a data item from a file: @@ -170,6 +262,53 @@ metadata: type: Opaque ``` +Like ConfigMaps, generated Secrets can be used in Deployments by refering to the name of the secretGenerator: + +```shell +# Create a password.txt file +cat <./password.txt +username=admin +password=secret +EOF + +cat <deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + labels: + app: my-app +spec: + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: app + image: my-app + volumeMount: + - name: password + mountPath: /secrets + volumes: + - name: password + secret: + secretName: example-secret-1 +EOF + +cat <./kustomization.yaml +resources: +- deployment.yaml +secretGenerator: +- name: example-secret-1 + files: + - password.txt +EOF +``` + #### generatorOptions The generated ConfigMaps and Secrets have a content hash suffix appended. This ensures that a new ConfigMap or Secret is generated when the contents are changed. To disable the behavior of appending a suffix, one can use `generatorOptions`. Besides that, it is also possible to specify cross-cutting options for generated ConfigMaps and Secrets. From ea0c1bb04b02b290e2dd781697442991440c2696 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20L=C3=A9one?= Date: Fri, 26 Mar 2021 14:19:44 +0100 Subject: [PATCH 030/218] add infos about docker setups --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 8ec876eef2..20e960935b 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,8 @@ make container-serve Open up your browser to http://localhost:1313 to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh. +If you see errors, it probably means that the hugo container did not have enough computing resources available. To solve it, increase the amount of allowed CPU and memory usage for Docker on your machine ([MacOSX](https://docs.docker.com/docker-for-mac/#resources) and [Windows](https://docs.docker.com/docker-for-windows/#resources)). + ## Running the website locally using Hugo Make sure to install the Hugo extended version specified by the `HUGO_VERSION` environment variable in the [`netlify.toml`](netlify.toml#L10) file. From 6a3c783af66571e9daad1d9e61b9f9187f37a52f Mon Sep 17 00:00:00 2001 From: Andreas Deininger Date: Sat, 27 Mar 2021 20:59:28 +0100 Subject: [PATCH 031/218] Fixing typo. --- README-de.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-de.md b/README-de.md index b570f43671..b6f4491e70 100644 --- a/README-de.md +++ b/README-de.md @@ -9,7 +9,7 @@ Herzlich willkommen! Dieses Repository enthält alle Assets, die zur Erstellung Sie können auf die Schaltfläche **Fork** im oberen rechten Bereich des Bildschirms klicken, um eine Kopie dieses Repositorys in Ihrem GitHub-Konto zu erstellen. Diese Kopie wird als *Fork* bezeichnet. Nehmen Sie die gewünschten Änderungen an Ihrem Fork vor. Wenn Sie bereit sind, diese Änderungen an uns zu senden, gehen Sie zu Ihrem Fork und erstellen Sie eine neue Pull-Anforderung, um uns darüber zu informieren. -Sobald Ihre Pull-Anfrage erstellt wurde, übernimmt ein Rezensent von Kubernetes die Verantwortung für klares, umsetzbares Feedback. Als Eigentümer des Pull-Request **liegt es in Ihrer Verantwortung Ihren Pull-Reqest enstsprechend des Feedbacks, dass Sie vom Kubernetes-Reviewer erhalten haben abzuändern.** Beachten Sie auch, dass Sie am Ende mehr als einen Rezensenten von Kubernetes erhalten, der Ihnen Feedback gibt, oder dass Sie Rückmeldungen von einem Rezensenten von Kubernetes erhalten, der sich von demjenigen unterscheidet, der ursprünglich für das Feedback zugewiesen wurde. In einigen Fällen kann es vorkommen, dass einer Ihrer Prüfer bei Bedarf eine technische Überprüfung von einem [Kubernetes Tech-Reviewer](https://github.com/kubernetes/website/wiki/tech-reviewers) anfordert. Reviewer geben ihr Bestes, um zeitnah Feedback zu geben, die Antwortzeiten können jedoch je nach den Umständen variieren. +Sobald Ihre Pull-Anfrage erstellt wurde, übernimmt ein Rezensent von Kubernetes die Verantwortung für klares, umsetzbares Feedback. Als Eigentümer des Pull-Request **liegt es in Ihrer Verantwortung Ihren Pull-Reqest entsprechend des Feedbacks, dass Sie vom Kubernetes-Reviewer erhalten haben abzuändern.** Beachten Sie auch, dass Sie am Ende mehr als einen Rezensenten von Kubernetes erhalten, der Ihnen Feedback gibt, oder dass Sie Rückmeldungen von einem Rezensenten von Kubernetes erhalten, der sich von demjenigen unterscheidet, der ursprünglich für das Feedback zugewiesen wurde. In einigen Fällen kann es vorkommen, dass einer Ihrer Prüfer bei Bedarf eine technische Überprüfung von einem [Kubernetes Tech-Reviewer](https://github.com/kubernetes/website/wiki/tech-reviewers) anfordert. Reviewer geben ihr Bestes, um zeitnah Feedback zu geben, die Antwortzeiten können jedoch je nach den Umständen variieren. Weitere Informationen zum Beitrag zur Kubernetes-Dokumentation finden Sie unter: From 194fe9953267e6049bfb7101e49c1346eab0e897 Mon Sep 17 00:00:00 2001 From: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com> Date: Fri, 26 Mar 2021 14:59:24 +0900 Subject: [PATCH 032/218] Update concepts/storage/volumes.md --- content/en/docs/concepts/storage/volumes.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index 8ed91067e0..4c944b6909 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -33,10 +33,9 @@ drivers, but the functionality is somewhat limited. Kubernetes supports many types of volumes. A {{< glossary_tooltip term_id="pod" text="Pod" >}} can use any number of volume types simultaneously. Ephemeral volume types have a lifetime of a pod, but persistent volumes exist beyond -the lifetime of a pod. Consequently, a volume outlives any containers -that run within the pod, and data is preserved across container restarts. When a pod -ceases to exist, Kubernetes destroys ephemeral volumes; however, Kubernetes does not -destroy persistent volumes. +the lifetime of a pod. When a pod ceases to exist, Kubernetes destroys ephemeral volumes; +however, Kubernetes does not destroy persistent volumes. +For any kind of volume in a given pod, data is preserved across container restarts. At its core, a volume is just a directory, possibly with some data in it, which is accessible to the containers in a pod. How that directory comes to be, the From a67181d4290d87bb186bdf2f28e3802dc3eefc79 Mon Sep 17 00:00:00 2001 From: huyuqing Date: Mon, 29 Mar 2021 16:43:04 +0800 Subject: [PATCH 033/218] Fix line separation in concepts/architecture/nodes --- content/de/docs/concepts/architecture/nodes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/de/docs/concepts/architecture/nodes.md b/content/de/docs/concepts/architecture/nodes.md index b790e68035..e717885339 100644 --- a/content/de/docs/concepts/architecture/nodes.md +++ b/content/de/docs/concepts/architecture/nodes.md @@ -148,6 +148,7 @@ Wenn ein Node in einer Cloud-Umgebung ausgeführt wird und sich in einem schlech Der dritte ist die Überwachung des Zustands der Nodes. Der Node Controller ist dafür verantwortlich, die NodeReady-Bedingung von NodeStatus auf ConditionUnknown zu aktualisieren, wenn ein wenn ein Node unerreichbar wird (der Node Controller empfängt aus irgendeinem Grund keine Herzschläge mehr, z.B. weil der Node heruntergefahren ist) und später alle Pods aus dem Node zu entfernen (und diese ordnungsgemäss zu beenden), wenn der Node weiterhin unzugänglich ist. (Die Standard-Timeouts sind 40s, um ConditionUnknown zu melden und 5 Minuten, um mit der Evakuierung der Pods zu beginnen). + Der Node Controller überprüft den Zustand jedes Nodes alle `--node-monitor-period` Sekunden. From bf2cfd8dc303aeef50091b7777960ff0e72687d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20L=C3=A9one?= Date: Mon, 29 Mar 2021 14:36:23 +0200 Subject: [PATCH 034/218] Fix --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 20e960935b..d548cd32aa 100644 --- a/README.md +++ b/README.md @@ -43,10 +43,10 @@ make container-image make container-serve ``` -Open up your browser to http://localhost:1313 to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh. - If you see errors, it probably means that the hugo container did not have enough computing resources available. To solve it, increase the amount of allowed CPU and memory usage for Docker on your machine ([MacOSX](https://docs.docker.com/docker-for-mac/#resources) and [Windows](https://docs.docker.com/docker-for-windows/#resources)). +Open up your browser to http://localhost:1313 to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh. + ## Running the website locally using Hugo Make sure to install the Hugo extended version specified by the `HUGO_VERSION` environment variable in the [`netlify.toml`](netlify.toml#L10) file. From d7f272061672e0ac7fb1b242136238a40eb39730 Mon Sep 17 00:00:00 2001 From: wangyysde Date: Mon, 29 Mar 2021 23:06:56 +0800 Subject: [PATCH 035/218] Update system-metrics.md --- .../concepts/cluster-administration/system-metrics.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/zh/docs/concepts/cluster-administration/system-metrics.md b/content/zh/docs/concepts/cluster-administration/system-metrics.md index cc3cb44fa7..a7053da87d 100644 --- a/content/zh/docs/concepts/cluster-administration/system-metrics.md +++ b/content/zh/docs/concepts/cluster-administration/system-metrics.md @@ -22,7 +22,7 @@ System component metrics can give a better look into what is happening inside th Kubernetes components emit metrics in [Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/). This format is structured plain text, designed so that people and machines can both read it. --> -系统组件指标可以更好地了解系统内部发生的情况。指标对于构建仪表板和告警特别有用。 +通过系统组件指标可以更好地了解系统组个内部发生的情况。系统组件指标对于构建仪表板和告警特别有用。 Kubernetes 组件以 [Prometheus 格式](https://prometheus.io/docs/instrumenting/exposition_formats/) 生成度量值。 @@ -37,10 +37,10 @@ In most cases metrics are available on `/metrics` endpoint of the HTTP server. F Examples of those components: --> -## Kubernetes 中的指标 +## Kubernetes 中组件的指标 -在大多数情况下，可以在 HTTP 服务器的 `/metrics` 端点上访问度量值。 -对于默认情况下不公开端点的组件，可以使用 `--bind-address` 标志启用。 +在大多数情况下，可以通过HTTP访问组件endpoint的 `/metrics` 路径来获取组件的度量值。 +对于那些默认情况下不暴露的endpoint的组件，可以使用 `--bind-address` 标志启用。 这些组件的示例： From f6d44d37acd60cfdf417a4f6241331a16810077b Mon Sep 17 00:00:00 2001 From: ooeunz Date: Wed, 31 Mar 2021 16:56:27 +0900 Subject: [PATCH 036/218] Fixed broken link in content/en/blog/_posts/2019-11-26-cloud-native-java-controller-sdk.md --- .../blog/_posts/2019-11-26-cloud-native-java-controller-sdk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2019-11-26-cloud-native-java-controller-sdk.md b/content/en/blog/_posts/2019-11-26-cloud-native-java-controller-sdk.md index 2474d07a95..80926b6bc4 100644 --- a/content/en/blog/_posts/2019-11-26-cloud-native-java-controller-sdk.md +++ b/content/en/blog/_posts/2019-11-26-cloud-native-java-controller-sdk.md @@ -58,7 +58,7 @@ Take maven project as example, adding the following dependencies into your depen Then we can make use of the provided builder libraries to write your own controller. For example, the following one is a simple controller prints out node information -on watch notification, see complete example [here](https://github.com/kubernetes-client/java/blob/master/examples/src/main/java/io/kubernetes/client/examples/ControllerExample.java): +on watch notification, see complete example [here](https://github.com/kubernetes-client/java/blob/master/examples/examples-release-13/src/main/java/io/kubernetes/client/examples/ControllerExample.java): ```java ... From edd432194ff1d94e767939500a082f390d77bc89 Mon Sep 17 00:00:00 2001 From: huyuqing Date: Thu, 1 Apr 2021 10:16:56 +0800 Subject: [PATCH 037/218] Remove redundant characters in concepts/architecture/nodes --- content/de/docs/concepts/architecture/nodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/de/docs/concepts/architecture/nodes.md b/content/de/docs/concepts/architecture/nodes.md index e717885339..933346a4d5 100644 --- a/content/de/docs/concepts/architecture/nodes.md +++ b/content/de/docs/concepts/architecture/nodes.md @@ -147,7 +147,7 @@ Die zweite ist, die interne Node-Liste des Node Controllers mit der Liste der ve Wenn ein Node in einer Cloud-Umgebung ausgeführt wird und sich in einem schlechten Zustand befindet, fragt der Node Controller den Cloud-Anbieter, ob die virtuelle Maschine für diesen Node noch verfügbar ist. Wenn nicht, löscht der Node Controller den Node aus seiner Node-Liste. Der dritte ist die Überwachung des Zustands der Nodes. Der Node Controller ist dafür verantwortlich, -die NodeReady-Bedingung von NodeStatus auf ConditionUnknown zu aktualisieren, wenn ein wenn ein Node unerreichbar wird (der Node Controller empfängt aus irgendeinem Grund keine Herzschläge mehr, z.B. weil der Node heruntergefahren ist) und später alle Pods aus dem Node zu entfernen (und diese ordnungsgemäss zu beenden), wenn der Node weiterhin unzugänglich ist. (Die Standard-Timeouts sind 40s, um ConditionUnknown zu melden und 5 Minuten, um mit der Evakuierung der Pods zu beginnen). +die NodeReady-Bedingung von NodeStatus auf ConditionUnknown zu aktualisieren, wenn ein Node unerreichbar wird (der Node Controller empfängt aus irgendeinem Grund keine Herzschläge mehr, z.B. weil der Node heruntergefahren ist) und später alle Pods aus dem Node zu entfernen (und diese ordnungsgemäss zu beenden), wenn der Node weiterhin unzugänglich ist. (Die Standard-Timeouts sind 40s, um ConditionUnknown zu melden und 5 Minuten, um mit der Evakuierung der Pods zu beginnen). Der Node Controller überprüft den Zustand jedes Nodes alle `--node-monitor-period` Sekunden. From cfc66c0f85fa822841af54f6d75a3d5243ae339b Mon Sep 17 00:00:00 2001 From: Adhityaa Chandrasekar Date: Tue, 23 Mar 2021 14:12:01 +0000 Subject: [PATCH 038/218] blog: introducing suspended jobs Signed-off-by: Adhityaa Chandrasekar --- .../2021-04-12-introducing-suspended-jobs.md | 110 ++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 content/en/blog/_posts/2021-04-12-introducing-suspended-jobs.md diff --git a/content/en/blog/_posts/2021-04-12-introducing-suspended-jobs.md b/content/en/blog/_posts/2021-04-12-introducing-suspended-jobs.md new file mode 100644 index 0000000000..4d46453adf --- /dev/null +++ b/content/en/blog/_posts/2021-04-12-introducing-suspended-jobs.md @@ -0,0 +1,110 @@ +--- +title: "Introducing Suspended Jobs" +date: 2021-04-12 +slug: introducing-suspended-jobs +layout: blog +--- + +**Author:** Adhityaa Chandrasekar (Google) + +[Jobs](/docs/concepts/workloads/controllers/job/) are a crucial part of +Kubernetes API. While other kinds of workloads such as [Deployments](/docs/concepts/workloads/controllers/deployment/), +[ReplicaSets](/docs/concepts/workloads/controllers/replicaset/), +[StatefulSets](/docs/concepts/workloads/controllers/statefulset/), and +[DaemonSets](/docs/concepts/workloads/controllers/daemonset/) +solve use-cases that require Pods to run forever, Jobs are useful when Pods need +to run to completion. Commonly used in parallel batch processing, Jobs can be +used in a variety of applications ranging from video rendering and database +maintenance to sending bulk emails and scientific computing. + +While the amount of parallelism and the conditions for Job completion are +configurable, the Kubernetes API lacked the ability to suspend and resume Jobs. +This is often desired when cluster resources are limited and a higher priority +Job needs to execute in the place of another Job. Deleting the lower priority +Job is a poor workaround as Pod completion history and other metrics associated +with the Job will be lost. + +With the recent Kubernetes 1.21 release, you will be able to suspend a Job by +updating its spec. The feature is currently in **alpha** and requires you to +enable the `SuspendJob` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/) +and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/) +in order to use it. + +## API changes + +A new boolean field `suspend` is introduced in the Job spec API. Let's say I +create the following Job: + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: my-job +spec: + suspend: true + parallelism: 2 + completions: 10 + template: + spec: + containers: + - name: my-container + image: busybox + command: ["sleep", "5"] + restartPolicy: Never +``` + +Jobs are not suspended by default, so I'm explicitly setting the `suspend` field +to true in the above Job spec. In the above example, the Job controller will +refrain from creating Pods until I'm ready to start the Job, which I can do by +updating the field to false. + +As another example, consider a Job that was created with the `suspend` field +omitted. The Job controller will happily create Pods to work towards Job +completion. However, before the Job completes, if I explicitly set the field to +true with a Job update, the Job controller will terminate all active Pods that +are running and will wait indefinitely for the flag to be flipped back to false. +Pod termination is done by sending a SIGTERM signal to all active Pods; the +[graceful termination period](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) +defined in the Pod spec will be honoured. Pods terminated this way will not be +counted as failures by the Job controller. + +It is important to understand that succeeded and failed Pods from the past will +continue to exist after you suspend a Job. That is, that they will count towards +Job completion once you resume it. You can verify this by looking at Job's +status before and after suspension. + +Read the [documentation](/docs/concepts/workloads/controllers/job#suspending-a-job) +for a full overview of this new feature. + +## Where is this useful? + +Let's say I'm the operator of a large cluster. I have many users submitting Jobs +to the cluster, but not all Jobs are created equal — some Jobs are more +important than others. Cluster resources aren't infinite either, so all users +must share resources. If all Jobs were created in the suspended state and placed +in a pending queue, I can achieve priority-based Job scheduling by resuming Jobs +in the right order. + +As another motivational use-case, consider a cloud provider where compute +resources are cheaper at night than in the morning. If I have a long-running Job +that takes multiple days to complete, being able to suspend the Job in the +morning and then resume it in the evening every day can reduce costs. + +Since this field is a part of the Job spec, CronJobs automatically get this +feature for free too. + +## References and next steps + +If you're interested in a deeper dive into the rationale behind this feature and +the decisions we have taken, consider reading the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2232-suspend-jobs). +There's more detail on suspending and resuming jobs in the documentation for [Job](/docs/concepts/workloads/controllers/job#suspending-a-job). + +As previously mentioned, this feature is currently in alpha and is available +only if you explicitly opt-in through the `SuspendJob` feature gate. If this is +a feature you're interested in, please consider testing suspended Jobs in your +cluster and providing feedback. You can discuss this enhancement [on GitHub](https://github.com/kubernetes/enhancements/issues/2232). +The SIG Apps community also [meets regularly](https://github.com/kubernetes/community/tree/master/sig-apps#meetings) +and can be reached through [Slack or the mailing list](https://github.com/kubernetes/community/tree/master/sig-apps#contact). +Barring any unexpected changes to the API, we intend to graduate the feature to +beta in Kubernetes 1.22, so that the feature becomes available by default. From a68cd7345ff193d99893560a64871834450d67ac Mon Sep 17 00:00:00 2001 From: Soichiro KAWAMURA Date: Fri, 2 Apr 2021 20:20:49 +0900 Subject: [PATCH 039/218] fix anchor in URLs --- .../ja/docs/concepts/configuration/pod-priority-preemption.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/ja/docs/concepts/configuration/pod-priority-preemption.md b/content/ja/docs/concepts/configuration/pod-priority-preemption.md index b83d1c6867..5aa3cf42ca 100644 --- a/content/ja/docs/concepts/configuration/pod-priority-preemption.md +++ b/content/ja/docs/concepts/configuration/pod-priority-preemption.md @@ -110,7 +110,7 @@ globalDefault: false description: "この優先度クラスは他のPodをプリエンプトさせません。" ``` -## Podの優先度 +## Podの優先度 {#pod-priority} 一つ以上のPriorityClassがあれば、仕様にPriorityClassを指定したPodを作成することができるようになります。優先度のアドミッションコントローラーは`priorityClassName`フィールドを使用し、優先度の整数値を設定します。PriorityClassが見つからない場合、そのPodの作成は拒否されます。 @@ -237,4 +237,4 @@ kubeletによるリソース不足時のPodの追い出しでは、リソース ## {{% heading "whatsnext" %}} -* PriorityClassと関連付けてResourceQuotaを使用することに関して [デフォルトで優先度クラスの消費を制限する](/ja/docs/concepts/policy/resource-quotas/#limit-priority-class-consumption-by-default) +* PriorityClassと関連付けてResourceQuotaを使用することに関して [デフォルトで優先度クラスの消費を制限する](/ja/docs/concepts/policy/resource-quotas/#デフォルトで優先度クラスの消費を制限する) From 5401a7ae7563c69dab504591d28ab8e604b59167 Mon Sep 17 00:00:00 2001 From: TAKAHASHI Shuuji Date: Fri, 2 Apr 2021 15:25:47 +0000 Subject: [PATCH 040/218] Copy files to translate into Japanese --- content/ja/docs/tutorials/clusters/_index.md | 5 + .../ja/docs/tutorials/clusters/apparmor.md | 473 ++++++++++++++++++ 2 files changed, 478 insertions(+) create mode 100755 content/ja/docs/tutorials/clusters/_index.md create mode 100644 content/ja/docs/tutorials/clusters/apparmor.md diff --git a/content/ja/docs/tutorials/clusters/_index.md b/content/ja/docs/tutorials/clusters/_index.md new file mode 100755 index 0000000000..2db6156b6f --- /dev/null +++ b/content/ja/docs/tutorials/clusters/_index.md @@ -0,0 +1,5 @@ +--- +title: "Clusters" +weight: 60 +--- + diff --git a/content/ja/docs/tutorials/clusters/apparmor.md b/content/ja/docs/tutorials/clusters/apparmor.md new file mode 100644 index 0000000000..54c8a0f44c --- /dev/null +++ b/content/ja/docs/tutorials/clusters/apparmor.md @@ -0,0 +1,473 @@ +--- +reviewers: +- stclair +title: Restrict a Container's Access to Resources with AppArmor +content_type: tutorial +weight: 10 +--- + + + +{{< feature-state for_k8s_version="v1.4" state="beta" >}} + + +AppArmor is a Linux kernel security module that supplements the standard Linux user and group based +permissions to confine programs to a limited set of resources. AppArmor can be configured for any +application to reduce its potential attack surface and provide greater in-depth defense. It is +configured through profiles tuned to allow the access needed by a specific program or container, +such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either +*enforcing* mode, which blocks access to disallowed resources, or *complain* mode, which only reports +violations. + +AppArmor can help you to run a more secure deployment by restricting what containers are allowed to +do, and/or provide better auditing through system logs. However, it is important to keep in mind +that AppArmor is not a silver bullet and can only do so much to protect against exploits in your +application code. It is important to provide good, restrictive profiles, and harden your +applications and cluster from other angles as well. + + + +## {{% heading "objectives" %}} + + +* See an example of how to load a profile on a node +* Learn how to enforce the profile on a Pod +* Learn how to check that the profile is loaded +* See what happens when a profile is violated +* See what happens when a profile cannot be loaded + + + +## {{% heading "prerequisites" %}} + + +Make sure: + +1. Kubernetes version is at least v1.4 -- Kubernetes support for AppArmor was added in + v1.4. Kubernetes components older than v1.4 are not aware of the new AppArmor annotations, and + will **silently ignore** any AppArmor settings that are provided. To ensure that your Pods are + receiving the expected protections, it is important to verify the Kubelet version of your nodes: + + ```shell + kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.kubeletVersion}\n{end}' + ``` + ``` + gke-test-default-pool-239f5d02-gyn2: v1.4.0 + gke-test-default-pool-239f5d02-x1kf: v1.4.0 + gke-test-default-pool-239f5d02-xwux: v1.4.0 + ``` + +2. AppArmor kernel module is enabled -- For the Linux kernel to enforce an AppArmor profile, the + AppArmor kernel module must be installed and enabled. Several distributions enable the module by + default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the + module is enabled, check the `/sys/module/apparmor/parameters/enabled` file: + + ```shell + cat /sys/module/apparmor/parameters/enabled + Y + ``` + + If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor + options if the kernel module is not enabled. + + {{< note >}} + Ubuntu carries many AppArmor patches that have not been merged into the upstream Linux + kernel, including patches that add additional hooks and features. Kubernetes has only been + tested with the upstream version, and does not promise support for other features. + {{< /note >}} + +3. Container runtime supports AppArmor -- Currently all common Kubernetes-supported container + runtimes should support AppArmor, like {{< glossary_tooltip term_id="docker">}}, + {{< glossary_tooltip term_id="cri-o" >}} or {{< glossary_tooltip term_id="containerd" >}}. + Please refer to the corresponding runtime documentation and verify that the cluster fulfills + the requirements to use AppArmor. + +4. Profile is loaded -- AppArmor is applied to a Pod by specifying an AppArmor profile that each + container should be run with. If any of the specified profiles is not already loaded in the + kernel, the Kubelet (>= v1.4) will reject the Pod. You can view which profiles are loaded on a + node by checking the `/sys/kernel/security/apparmor/profiles` file. For example: + + ```shell + ssh gke-test-default-pool-239f5d02-gyn2 "sudo cat /sys/kernel/security/apparmor/profiles | sort" + ``` + ``` + apparmor-test-deny-write (enforce) + apparmor-test-audit-write (enforce) + docker-default (enforce) + k8s-nginx (enforce) + ``` + + For more details on loading profiles on nodes, see + [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + +As long as the Kubelet version includes AppArmor support (>= v1.4), the Kubelet will reject a Pod +with AppArmor options if any of the prerequisites are not met. You can also verify AppArmor support +on nodes by checking the node ready condition message (though this is likely to be removed in a +later release): + +```shell +kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}\n{end}' +``` +``` +gke-test-default-pool-239f5d02-gyn2: kubelet is posting ready status. AppArmor enabled +gke-test-default-pool-239f5d02-x1kf: kubelet is posting ready status. AppArmor enabled +gke-test-default-pool-239f5d02-xwux: kubelet is posting ready status. AppArmor enabled +``` + + + + + +## Securing a Pod + +{{< note >}} +AppArmor is currently in beta, so options are specified as annotations. Once support graduates to +general availability, the annotations will be replaced with first-class fields (more details in +[Upgrade path to GA](#upgrade-path-to-general-availability)). +{{< /note >}} + +AppArmor profiles are specified *per-container*. To specify the AppArmor profile to run a Pod +container with, add an annotation to the Pod's metadata: + +```yaml +container.apparmor.security.beta.kubernetes.io/: +``` + +Where `` is the name of the container to apply the profile to, and `` +specifies the profile to apply. The `profile_ref` can be one of: + +* `runtime/default` to apply the runtime's default profile +* `localhost/` to apply the profile loaded on the host with the name `` +* `unconfined` to indicate that no profiles will be loaded + +See the [API Reference](#api-reference) for the full details on the annotation and profile name formats. + +Kubernetes AppArmor enforcement works by first checking that all the prerequisites have been +met, and then forwarding the profile selection to the container runtime for enforcement. If the +prerequisites have not been met, the Pod will be rejected, and will not run. + +To verify that the profile was applied, you can look for the AppArmor security option listed in the container created event: + +```shell +kubectl get events | grep Created +``` +``` +22s 22s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet e2e-test-stclair-node-pool-31nt} Created container with docker id 269a53b202d3; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] +``` + +You can also verify directly that the container's root process is running with the correct profile by checking its proc attr: + +```shell +kubectl exec cat /proc/1/attr/current +``` +``` +k8s-apparmor-example-deny-write (enforce) +``` + +## Example + +*This example assumes you have already set up a cluster with AppArmor support.* + +First, we need to load the profile we want to use onto our nodes. This profile denies all file writes: + +```shell +#include + +profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { + #include + + file, + + # Deny all file writes. + deny /** w, +} +``` + +Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our +nodes. For this example we'll just use SSH to install the profiles, but other approaches are +discussed in [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + +```shell +NODES=( + # The SSH-accessible domain names of your nodes + gke-test-default-pool-239f5d02-gyn2.us-central1-a.my-k8s + gke-test-default-pool-239f5d02-x1kf.us-central1-a.my-k8s + gke-test-default-pool-239f5d02-xwux.us-central1-a.my-k8s) +for NODE in ${NODES[*]}; do ssh $NODE 'sudo apparmor_parser -q < + +profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { + #include + + file, + + # Deny all file writes. + deny /** w, +} +EOF' +done +``` + +Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: + +{{< codenew file="pods/security/hello-apparmor.yaml" >}} + +```shell +kubectl create -f ./hello-apparmor.yaml +``` + +If we look at the pod events, we can see that the Pod container was created with the AppArmor +profile "k8s-apparmor-example-deny-write": + +```shell +kubectl get events | grep hello-apparmor +``` +``` +14s 14s 1 hello-apparmor Pod Normal Scheduled {default-scheduler } Successfully assigned hello-apparmor to gke-test-default-pool-239f5d02-gyn2 +14s 14s 1 hello-apparmor Pod spec.containers{hello} Normal Pulling {kubelet gke-test-default-pool-239f5d02-gyn2} pulling image "busybox" +13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Pulled {kubelet gke-test-default-pool-239f5d02-gyn2} Successfully pulled image "busybox" +13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet gke-test-default-pool-239f5d02-gyn2} Created container with docker id 06b6cd1c0989; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] +13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Started {kubelet gke-test-default-pool-239f5d02-gyn2} Started container with docker id 06b6cd1c0989 +``` + +We can verify that the container is actually running with that profile by checking its proc attr: + +```shell +kubectl exec hello-apparmor cat /proc/1/attr/current +``` +``` +k8s-apparmor-example-deny-write (enforce) +``` + +Finally, we can see what happens if we try to violate the profile by writing to a file: + +```shell +kubectl exec hello-apparmor touch /tmp/test +``` +``` +touch: /tmp/test: Permission denied +error: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1 +``` + +To wrap up, let's look at what happens if we try to specify a profile that hasn't been loaded: + +```shell +kubectl create -f /dev/stdin < +Annotations: container.apparmor.security.beta.kubernetes.io/hello=localhost/k8s-apparmor-example-allow-write +Status: Pending +Reason: AppArmor +Message: Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded +IP: +Controllers: +Containers: + hello: + Container ID: + Image: busybox + Image ID: + Port: + Command: + sh + -c + echo 'Hello AppArmor!' && sleep 1h + State: Waiting + Reason: Blocked + Ready: False + Restart Count: 0 + Environment: + Mounts: + /var/run/secrets/kubernetes.io/serviceaccount from default-token-dnz7v (ro) +Conditions: + Type Status + Initialized True + Ready False + PodScheduled True +Volumes: + default-token-dnz7v: + Type: Secret (a volume populated by a Secret) + SecretName: default-token-dnz7v + Optional: false +QoS Class: BestEffort +Node-Selectors: +Tolerations: +Events: + FirstSeen LastSeen Count From SubobjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 23s 23s 1 {default-scheduler } Normal Scheduled Successfully assigned hello-apparmor-2 to e2e-test-stclair-node-pool-t1f5 + 23s 23s 1 {kubelet e2e-test-stclair-node-pool-t1f5} Warning AppArmor Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded +``` + +Note the pod status is Failed, with a helpful error message: `Pod Cannot enforce AppArmor: profile +"k8s-apparmor-example-allow-write" is not loaded`. An event was also recorded with the same message. + +## Administration + +### Setting up nodes with profiles + +Kubernetes does not currently provide any native mechanisms for loading AppArmor profiles onto +nodes. There are lots of ways to setup the profiles though, such as: + +* Through a [DaemonSet](/docs/concepts/workloads/controllers/daemonset/) that runs a Pod on each node to + ensure the correct profiles are loaded. An example implementation can be found + [here](https://git.k8s.io/kubernetes/test/images/apparmor-loader). +* At node initialization time, using your node initialization scripts (e.g. Salt, Ansible, etc.) or + image. +* By copying the profiles to each node and loading them through SSH, as demonstrated in the + [Example](#example). + +The scheduler is not aware of which profiles are loaded onto which node, so the full set of profiles +must be loaded onto every node. An alternative approach is to add a node label for each profile (or +class of profiles) on the node, and use a +[node selector](/docs/concepts/scheduling-eviction/assign-pod-node/) to ensure the Pod is run on a +node with the required profile. + +### Restricting profiles with the PodSecurityPolicy + +If the PodSecurityPolicy extension is enabled, cluster-wide AppArmor restrictions can be applied. To +enable the PodSecurityPolicy, the following flag must be set on the `apiserver`: + +``` +--enable-admission-plugins=PodSecurityPolicy[,others...] +``` + +The AppArmor options can be specified as annotations on the PodSecurityPolicy: + +```yaml +apparmor.security.beta.kubernetes.io/defaultProfileName: +apparmor.security.beta.kubernetes.io/allowedProfileNames: [,others...] +``` + +The default profile name option specifies the profile to apply to containers by default when none is +specified. The allowed profile names option specifies a list of profiles that Pod containers are +allowed to be run with. If both options are provided, the default must be allowed. The profiles are +specified in the same format as on containers. See the [API Reference](#api-reference) for the full +specification. + +### Disabling AppArmor + +If you do not want AppArmor to be available on your cluster, it can be disabled by a command-line flag: + +``` +--feature-gates=AppArmor=false +``` + +When disabled, any Pod that includes an AppArmor profile will fail validation with a "Forbidden" +error. Note that by default docker always enables the "docker-default" profile on non-privileged +pods (if the AppArmor kernel module is enabled), and will continue to do so even if the feature-gate +is disabled. The option to disable AppArmor will be removed when AppArmor graduates to general +availability (GA). + +### Upgrading to Kubernetes v1.4 with AppArmor + +No action is required with respect to AppArmor to upgrade your cluster to v1.4. However, if any +existing pods had an AppArmor annotation, they will not go through validation (or PodSecurityPolicy +admission). If permissive profiles are loaded on the nodes, a malicious user could pre-apply a +permissive profile to escalate the pod privileges above the docker-default. If this is a concern, it +is recommended to scrub the cluster of any pods containing an annotation with +`apparmor.security.beta.kubernetes.io`. + +### Upgrade path to General Availability + +When AppArmor is ready to be graduated to general availability (GA), the options currently specified +through annotations will be converted to fields. Supporting all the upgrade and downgrade paths +through the transition is very nuanced, and will be explained in detail when the transition +occurs. We will commit to supporting both fields and annotations for at least 2 releases, and will +explicitly reject the annotations for at least 2 releases after that. + +## Authoring Profiles + +Getting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some +tools to help with that: + +* `aa-genprof` and `aa-logprof` generate profile rules by monitoring an application's activity and + logs, and admitting the actions it takes. Further instructions are provided by the + [AppArmor documentation](https://gitlab.com/apparmor/apparmor/wikis/Profiling_with_tools). +* [bane](https://github.com/jfrazelle/bane) is an AppArmor profile generator for Docker that uses a + simplified profile language. + +It is recommended to run your application through Docker on a development workstation to generate +the profiles, but there is nothing preventing running the tools on the Kubernetes node where your +Pod is running. + +To debug problems with AppArmor, you can check the system logs to see what, specifically, was +denied. AppArmor logs verbose messages to `dmesg`, and errors can usually be found in the system +logs or through `journalctl`. More information is provided in +[AppArmor failures](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Failures). + + +## API Reference + +### Pod Annotation + +Specifying the profile a container will run with: + +- **key**: `container.apparmor.security.beta.kubernetes.io/` + Where `` matches the name of a container in the Pod. + A separate profile can be specified for each container in the Pod. +- **value**: a profile reference, described below + +### Profile Reference + +- `runtime/default`: Refers to the default runtime profile. + - Equivalent to not specifying a profile (without a PodSecurityPolicy default), except it still + requires AppArmor to be enabled. + - For Docker, this resolves to the + [`docker-default`](https://docs.docker.com/engine/security/apparmor/) profile for non-privileged + containers, and unconfined (no profile) for privileged containers. +- `localhost/`: Refers to a profile loaded on the node (localhost) by name. + - The possible profile names are detailed in the + [core policy reference](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Core_Policy_Reference#profile-names-and-attachment-specifications). +- `unconfined`: This effectively disables AppArmor on the container. + +Any other profile reference format is invalid. + +### PodSecurityPolicy Annotations + +Specifying the default profile to apply to containers when none is provided: + +* **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` +* **value**: a profile reference, described above + +Specifying the list of profiles Pod containers is allowed to specify: + +* **key**: `apparmor.security.beta.kubernetes.io/allowedProfileNames` +* **value**: a comma-separated list of profile references (described above) + - Although an escaped comma is a legal character in a profile name, it cannot be explicitly + allowed here. + + + +## {{% heading "whatsnext" %}} + + +Additional resources: + +* [Quick guide to the AppArmor profile language](https://gitlab.com/apparmor/apparmor/wikis/QuickProfileLanguage) +* [AppArmor core policy reference](https://gitlab.com/apparmor/apparmor/wikis/Policy_Layout) + + From 5d0031cbdd96a41063e6860595cf923b7fee2e20 Mon Sep 17 00:00:00 2001 From: TAKAHASHI Shuuji Date: Fri, 2 Apr 2021 15:25:47 +0000 Subject: [PATCH 041/218] Translate _index.md into Japanese --- content/ja/docs/tutorials/clusters/_index.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/ja/docs/tutorials/clusters/_index.md b/content/ja/docs/tutorials/clusters/_index.md index 2db6156b6f..0f3a08fdda 100755 --- a/content/ja/docs/tutorials/clusters/_index.md +++ b/content/ja/docs/tutorials/clusters/_index.md @@ -1,5 +1,4 @@ --- -title: "Clusters" +title: "クラスター" weight: 60 --- - From e3d9f76bff9686141cec2faab0158a41235d667b Mon Sep 17 00:00:00 2001 From: TAKAHASHI Shuuji Date: Fri, 2 Apr 2021 15:25:47 +0000 Subject: [PATCH 042/218] Translate tutorials/clusters/apparmor into Japanese --- .../ja/docs/tutorials/clusters/apparmor.md | 266 ++++++------------ 1 file changed, 84 insertions(+), 182 deletions(-) diff --git a/content/ja/docs/tutorials/clusters/apparmor.md b/content/ja/docs/tutorials/clusters/apparmor.md index 54c8a0f44c..6f5bd08709 100644 --- a/content/ja/docs/tutorials/clusters/apparmor.md +++ b/content/ja/docs/tutorials/clusters/apparmor.md @@ -1,7 +1,5 @@ --- -reviewers: -- stclair -title: Restrict a Container's Access to Resources with AppArmor +title: AppArmorを使用してコンテナのリソースへのアクセスを制限する content_type: tutorial weight: 10 --- @@ -10,43 +8,23 @@ weight: 10 {{< feature-state for_k8s_version="v1.4" state="beta" >}} +AppArmorは、Linux標準のユーザー・グループをベースとしたパーミッションを補完するLinuxカーネルのセキュリティモジュールであり、プログラムのアクセスを限定されたリソースセットに制限するために利用されます。AppArmorを設定することで、任意のアプリケーションの攻撃サーフェイスとなりうる面を減らしたり、より優れた多重の防御を提供できます。AppArmorは、たとえばLinuxのcapability、ネットワークアクセス、ファイルのパーミッションなど、特定のプログラムやコンテナに必要なアクセスを許可するようにチューニングされたプロファイルにより設定を行います。各プロファイルは、許可されなかったリソースへのアクセスをブロックする*enforcing*モードと、ルール違反を報告するだけの*complain*モードのいずれかで実行できます。 -AppArmor is a Linux kernel security module that supplements the standard Linux user and group based -permissions to confine programs to a limited set of resources. AppArmor can be configured for any -application to reduce its potential attack surface and provide greater in-depth defense. It is -configured through profiles tuned to allow the access needed by a specific program or container, -such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either -*enforcing* mode, which blocks access to disallowed resources, or *complain* mode, which only reports -violations. - -AppArmor can help you to run a more secure deployment by restricting what containers are allowed to -do, and/or provide better auditing through system logs. However, it is important to keep in mind -that AppArmor is not a silver bullet and can only do so much to protect against exploits in your -application code. It is important to provide good, restrictive profiles, and harden your -applications and cluster from other angles as well. - - +AppArmorを利用すれば、コンテナに許可することを制限したりシステムログを通してよりよい監査を提供することで、デプロイをよりセキュアにする助けになります。しかし、AppArmorは銀の弾丸ではなく、アプリケーションコードの悪用からの防御を強化できるだけであることを心に留めておくことが重要です。制限の強い優れたプロファイルを提供し、アプリケーションとクラスターを別の角度から強化することが重要です。 ## {{% heading "objectives" %}} - -* See an example of how to load a profile on a node -* Learn how to enforce the profile on a Pod -* Learn how to check that the profile is loaded -* See what happens when a profile is violated -* See what happens when a profile cannot be loaded - - +* プロファイルをノードに読み込む方法の例を見る +* Pod上でプロファイルを矯正する方法を学ぶ +* プロファイルが読み込まれたかを確認する方法を学ぶ +* プロファイルに違反した場合に何が起こるのかを見る +* プロファイルが読み込めなかった場合に何が起こるのかを見る ## {{% heading "prerequisites" %}} +以下のことを確認してください。 -Make sure: - -1. Kubernetes version is at least v1.4 -- Kubernetes support for AppArmor was added in - v1.4. Kubernetes components older than v1.4 are not aware of the new AppArmor annotations, and - will **silently ignore** any AppArmor settings that are provided. To ensure that your Pods are - receiving the expected protections, it is important to verify the Kubelet version of your nodes: +1. Kubernetesのバージョンがv1.4以上であること。KubernetesのAppArmorのサポートはv1.4で追加されました。v1.4より古いバージョンのKubernetesのコンポーネントは、新しいAppArmorのアノテーションを認識できないため、AppArmorの設定を与えたとしても**黙って無視されてしまいます**。Podが期待した保護を確実に受けられるようにするためには、次のようにノードのKubeletのバージョンを確認することが重要です。 ```shell kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.kubeletVersion}\n{end}' @@ -57,35 +35,22 @@ Make sure: gke-test-default-pool-239f5d02-xwux: v1.4.0 ``` -2. AppArmor kernel module is enabled -- For the Linux kernel to enforce an AppArmor profile, the - AppArmor kernel module must be installed and enabled. Several distributions enable the module by - default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the - module is enabled, check the `/sys/module/apparmor/parameters/enabled` file: +2. AppArmorカーネルモジュールが有効であること。LinuxカーネルがAppArmorプロファイルを強制するためには、AppArmorカーネルモジュールのインストールと有効化が必須です。UbuntuやSUSEなどのディストリビューションではデフォルトで有効化されますが、他の多くのディストリビューションでのサポートはオプションです。モジュールが有効になっているかチェックするには、次のように`/sys/module/apparmor/parameters/enabled`ファイルを確認します。 ```shell cat /sys/module/apparmor/parameters/enabled Y ``` - If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor - options if the kernel module is not enabled. + KubeletがAppArmorをサポートしていれば(>= v1.4)、カーネルモジュールが有効になっていない場合にはAppArmorオプションが付いたPodを拒否します。 {{< note >}} - Ubuntu carries many AppArmor patches that have not been merged into the upstream Linux - kernel, including patches that add additional hooks and features. Kubernetes has only been - tested with the upstream version, and does not promise support for other features. + UbuntuはAppArmorに対して、アップストリームのLinuxにマージしていない多数のパッチを当てています。その中には、追加のフックや機能を加えるパッチも含まれます。Kubernetesはアップストリームのバージョンでのみテストされており、その他の機能に対するサポートを約束していません。 {{< /note >}} -3. Container runtime supports AppArmor -- Currently all common Kubernetes-supported container - runtimes should support AppArmor, like {{< glossary_tooltip term_id="docker">}}, - {{< glossary_tooltip term_id="cri-o" >}} or {{< glossary_tooltip term_id="containerd" >}}. - Please refer to the corresponding runtime documentation and verify that the cluster fulfills - the requirements to use AppArmor. +3. コンテナランタイムがAppArmorをサポートしていること。現在、Kubernetesがサポートするすべての一般的なコンテナランタイム、{{< glossary_tooltip term_id="docker">}}、{{< glossary_tooltip term_id="cri-o" >}}、{{< glossary_tooltip term_id="containerd" >}}などは、AppArmorをサポートしています。関連するランタイムのドキュメントを参照して、クラスターがAppArmorを利用するための要求を満たしているかどうかを検証してください。 -4. Profile is loaded -- AppArmor is applied to a Pod by specifying an AppArmor profile that each - container should be run with. If any of the specified profiles is not already loaded in the - kernel, the Kubelet (>= v1.4) will reject the Pod. You can view which profiles are loaded on a - node by checking the `/sys/kernel/security/apparmor/profiles` file. For example: +4. プロファイルが読み込まれていること。AppArmorがPodに適用されるのは、各コンテナが実行されるべきAppArmorプロファイルを指定したときです。もし指定されたプロファイルがまだカーネルに読み込まれていなければ、Kubelet(>= v1.4)はPodを拒否します。どのプロファイルがノードに読み込まれているのかを確かめるには、次のようなコマンドを実行して`/sys/kernel/security/apparmor/profiles`をチェックします。 ```shell ssh gke-test-default-pool-239f5d02-gyn2 "sudo cat /sys/kernel/security/apparmor/profiles | sort" @@ -97,13 +62,9 @@ Make sure: k8s-nginx (enforce) ``` - For more details on loading profiles on nodes, see - [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + ノード上でのプロファイルの読み込みの詳細については、[プロファイルを使用したノードのセットアップ](#setting-up-nodes-with-profiles)を参照してください。 -As long as the Kubelet version includes AppArmor support (>= v1.4), the Kubelet will reject a Pod -with AppArmor options if any of the prerequisites are not met. You can also verify AppArmor support -on nodes by checking the node ready condition message (though this is likely to be removed in a -later release): +KubeletのバージョンがAppArmorサポートに対応しているもの(>= v1.4)である限り、Kubeletは必要条件を1つでも満たさないAppArmorオプションが付けられたPodをリジェクトします。また、ノード上のAppArmorのサポートは、次のようにready conditionのメッセージで確認することもできます(ただし、この機能は将来のリリースで削除される可能性があります)。 ```shell kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}\n{end}' @@ -114,39 +75,31 @@ gke-test-default-pool-239f5d02-x1kf: kubelet is posting ready status. AppArmor e gke-test-default-pool-239f5d02-xwux: kubelet is posting ready status. AppArmor enabled ``` - - -## Securing a Pod +## Podをセキュアにする {{< note >}} -AppArmor is currently in beta, so options are specified as annotations. Once support graduates to -general availability, the annotations will be replaced with first-class fields (more details in -[Upgrade path to GA](#upgrade-path-to-general-availability)). +AppArmorは現在beta版であるため、オプションはアノテーションとして指定します。将来サポートが一般利用可能(GA)になれば、アノテーションは第1級のフィールドで置き換えられます(詳細については、[一般利用可能(General Availability)への更新パス](#upgrade-path-to-general-availability)を参照してください)。 {{< /note >}} -AppArmor profiles are specified *per-container*. To specify the AppArmor profile to run a Pod -container with, add an annotation to the Pod's metadata: +AppArmorのプロファイルは*各コンテナごとに*指定します。Podのコンテナで実行するAppArmorのプロファイルを指定するには、Podのメタデータに次のようなアノテーションを追加します。 ```yaml container.apparmor.security.beta.kubernetes.io/: ``` -Where `` is the name of the container to apply the profile to, and `` -specifies the profile to apply. The `profile_ref` can be one of: +ここで、``はプロファイルを適用するコンテナの名前であり、``には適用するプロファイルを指定します。`profile_ref`は次の値のうち1つを指定します。 -* `runtime/default` to apply the runtime's default profile -* `localhost/` to apply the profile loaded on the host with the name `` -* `unconfined` to indicate that no profiles will be loaded +* `runtime/default`: ランタイムのデフォルトのプロファイルを適用する +* `localhost/`: ``という名前でホストにロードされたプロファイルを適用する +* `unconfined`: いかなるプロファイルもロードされないことを示す -See the [API Reference](#api-reference) for the full details on the annotation and profile name formats. +アノテーションとプロファイルの名前のフォーマットの詳細については、[APIリファレンス](#api-reference)を参照してください。 -Kubernetes AppArmor enforcement works by first checking that all the prerequisites have been -met, and then forwarding the profile selection to the container runtime for enforcement. If the -prerequisites have not been met, the Pod will be rejected, and will not run. +KubernetesのAppArmorの強制では、まずはじめにすべての前提条件が満たされているかどうかをチェックします。その後、強制を行うためにプロファイルの選択をコンテナランタイムに委ねます。前提条件が満たされなかった場合、Podはリジェクトされ、実行されません。 -To verify that the profile was applied, you can look for the AppArmor security option listed in the container created event: +プロファイルが適用されたかどうか確認するには、AppArmor securityオプションがコンテナ作成イベントに一覧されているかどうかを確認します。 ```shell kubectl get events | grep Created @@ -155,7 +108,7 @@ kubectl get events | grep Created 22s 22s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet e2e-test-stclair-node-pool-31nt} Created container with docker id 269a53b202d3; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] ``` -You can also verify directly that the container's root process is running with the correct profile by checking its proc attr: +proc attrを調べることで、コンテナのルートプロセスが正しいプロファイルで実行されているかどうかを直接確認することもできます。 ```shell kubectl exec cat /proc/1/attr/current @@ -164,11 +117,11 @@ kubectl exec cat /proc/1/attr/current k8s-apparmor-example-deny-write (enforce) ``` -## Example +## 例 {#example} -*This example assumes you have already set up a cluster with AppArmor support.* +*この例は、クラスターがすでにAppArmorのサポート付きでセットアップ済みであることを前提としています。* -First, we need to load the profile we want to use onto our nodes. This profile denies all file writes: +まず、使用したいプロファイルをノード上に読み込む必要があります。このプロファイルは、すべてのファイル書き込みを拒否します。 ```shell #include @@ -183,13 +136,11 @@ profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { } ``` -Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our -nodes. For this example we'll just use SSH to install the profiles, but other approaches are -discussed in [Setting up nodes with profiles](#setting-up-nodes-with-profiles). +Podがどのノードにスケジュールされるかは予測できないため、プロファイルはすべてのノードに読み込ませる必要があります。この例では、単純にSSHを使ってプロファイルをインストールしますが、[プロファイルを使用したノードのセットアップ](#setting-up-nodes-with-profiles)では、他のアプローチについて議論しています。 ```shell NODES=( - # The SSH-accessible domain names of your nodes + # SSHでアクセス可能なノードのドメイン名 gke-test-default-pool-239f5d02-gyn2.us-central1-a.my-k8s gke-test-default-pool-239f5d02-x1kf.us-central1-a.my-k8s gke-test-default-pool-239f5d02-xwux.us-central1-a.my-k8s) @@ -208,7 +159,7 @@ EOF' done ``` -Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: +次に、deny-writeプロファイルを使用した単純な "Hello AppArmor" Podを実行します。 {{< codenew file="pods/security/hello-apparmor.yaml" >}} @@ -216,8 +167,7 @@ Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: kubectl create -f ./hello-apparmor.yaml ``` -If we look at the pod events, we can see that the Pod container was created with the AppArmor -profile "k8s-apparmor-example-deny-write": +Podイベントを確認すると、PodコンテナがAppArmorプロファイル "k8s-apparmor-example-deny-write" を使用して作成されたことがわかります。 ```shell kubectl get events | grep hello-apparmor @@ -230,7 +180,7 @@ kubectl get events | grep hello-apparmor 13s 13s 1 hello-apparmor Pod spec.containers{hello} Normal Started {kubelet gke-test-default-pool-239f5d02-gyn2} Started container with docker id 06b6cd1c0989 ``` -We can verify that the container is actually running with that profile by checking its proc attr: +コンテナがこのプロファイルで実際に実行されていることを確認するために、コンテナのproc attrをチェックします。 ```shell kubectl exec hello-apparmor cat /proc/1/attr/current @@ -239,7 +189,7 @@ kubectl exec hello-apparmor cat /proc/1/attr/current k8s-apparmor-example-deny-write (enforce) ``` -Finally, we can see what happens if we try to violate the profile by writing to a file: +最後に、ファイルへの書き込みを行おうとすることで、プロファイルに違反すると何が起こるか見てみましょう。 ```shell kubectl exec hello-apparmor touch /tmp/test @@ -249,7 +199,7 @@ touch: /tmp/test: Permission denied error: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1 ``` -To wrap up, let's look at what happens if we try to specify a profile that hasn't been loaded: +まとめとして、読み込まれていないプロファイルを指定しようとするとどうなるのか見てみましょう。 ```shell kubectl create -f /dev/stdin < apparmor.security.beta.kubernetes.io/allowedProfileNames: [,others...] ``` -The default profile name option specifies the profile to apply to containers by default when none is -specified. The allowed profile names option specifies a list of profiles that Pod containers are -allowed to be run with. If both options are provided, the default must be allowed. The profiles are -specified in the same format as on containers. See the [API Reference](#api-reference) for the full -specification. +defaultProfileNameオプションには、noneが指定された場合にコンテナにデフォルトで適用されるプロファイルを指定します。allowedProfileNamesオプションには、Podコンテナの実行が許可されるプロファイルのリストを指定します。両方のオプションが指定された場合、デフォルトは許可されなければいけません。プロファイルはコンテナ上で同じフォーマットで指定されます。完全な仕様については、[APIリファレンス](#api-reference)を参照してください。 -### Disabling AppArmor +### AppArmorの無効化 -If you do not want AppArmor to be available on your cluster, it can be disabled by a command-line flag: +クラスタ上でAppArmorを利用可能にしたくない場合、次のコマンドラインフラグで無効化できます。 ``` --feature-gates=AppArmor=false ``` -When disabled, any Pod that includes an AppArmor profile will fail validation with a "Forbidden" -error. Note that by default docker always enables the "docker-default" profile on non-privileged -pods (if the AppArmor kernel module is enabled), and will continue to do so even if the feature-gate -is disabled. The option to disable AppArmor will be removed when AppArmor graduates to general -availability (GA). +無効化すると、AppArmorプロファイルを含むPodは"Forbidden"エラーで検証に失敗します。ただし、デフォルトのdockerは非特権Pod上では"docker-default"というプロファイルを常に有効化し(AppArmorカーネルモジュールが有効である場合)、フィーチャーゲートで無効化したとしても有効化し続けることに注意してください。AppArmorを無効化するオプションは、AppArmorが一般利用(GA)になったときに削除される予定です。 -### Upgrading to Kubernetes v1.4 with AppArmor +### AppArmorを使用するKubernetes v1.4にアップグレードする -No action is required with respect to AppArmor to upgrade your cluster to v1.4. However, if any -existing pods had an AppArmor annotation, they will not go through validation (or PodSecurityPolicy -admission). If permissive profiles are loaded on the nodes, a malicious user could pre-apply a -permissive profile to escalate the pod privileges above the docker-default. If this is a concern, it -is recommended to scrub the cluster of any pods containing an annotation with -`apparmor.security.beta.kubernetes.io`. +クラスタをv1.4にアップグレードするために、AppArmorに関する操作は必要ありません。ただし、既存のPodがAppArmorのアノテーションを持っている場合、検証(またはPodSecurityPolicy admission)は行われません。もしpermissiveなプロファイルがノードに読み込まれていた場合、悪意のあるユーザーがPodの権限を上述のdocker-defaultより昇格させるために、permissiveなプロファイルを再適用する恐れがあります。これが問題となる場合、`apparmor.security.beta.kubernetes.io`のアノテーションを含むすべてのPodのクラスターをクリーンアップすることを推奨します。 -### Upgrade path to General Availability +### 一般利用可能(General Availability)への更新パス {#upgrade-path-to-general-availability} -When AppArmor is ready to be graduated to general availability (GA), the options currently specified -through annotations will be converted to fields. Supporting all the upgrade and downgrade paths -through the transition is very nuanced, and will be explained in detail when the transition -occurs. We will commit to supporting both fields and annotations for at least 2 releases, and will -explicitly reject the annotations for at least 2 releases after that. +AppArmorが一般利用可能(GA)になったとき、現在アノテーションで指定しているオプションはフィールドに変換されます。移行中のすべてのアップグレードとダウングレードの経路をサポートするのは非常に微妙であるため、以降が必要になったときに詳細に説明する予定です。最低2リリースの間はフィールドとアノテーションの両方がサポートされるようにする予定です。最低2リリースの後は、アノテーションは明示的に拒否されるようになります。 -## Authoring Profiles +## Profilesの作成 -Getting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some -tools to help with that: +AppArmorのプロファイルを正しく指定するのはやっかいな作業です。幸い、その作業を補助するツールがいくつかあります。 -* `aa-genprof` and `aa-logprof` generate profile rules by monitoring an application's activity and - logs, and admitting the actions it takes. Further instructions are provided by the - [AppArmor documentation](https://gitlab.com/apparmor/apparmor/wikis/Profiling_with_tools). -* [bane](https://github.com/jfrazelle/bane) is an AppArmor profile generator for Docker that uses a - simplified profile language. +* `aa-genprof`および`aa-logprof`は、アプリケーションの動作とログを監視することによりプロファイルのルールを生成します。詳しい説明については、[AppArmor documentation](https://gitlab.com/apparmor/apparmor/wikis/Profiling_with_tools)を参照してください。 +* [bane](https://github.com/jfrazelle/bane)は、Docker向けのAppArmorのプロファイル・ジェネレータです。簡略化されたプロファイル言語を使用しています。 -It is recommended to run your application through Docker on a development workstation to generate -the profiles, but there is nothing preventing running the tools on the Kubernetes node where your -Pod is running. +プロファイルの生成には、アプリケーションを開発用ワークステーション上でDockerで実行することを推奨します。しかし、実際にPodが実行されるKubernetesノード上でツールを実行してはいけない理由はありません。 -To debug problems with AppArmor, you can check the system logs to see what, specifically, was -denied. AppArmor logs verbose messages to `dmesg`, and errors can usually be found in the system -logs or through `journalctl`. More information is provided in -[AppArmor failures](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Failures). +AppArmorに関する問題をデバッグするには、システムログをチェックして、特に何が拒否されたのかを確認できます。AppArmorのログは`dmesg`にverboseメッセージを送り、エラーは通常システムログまたは`journalctl`で確認できます。詳しい情報は、[AppArmor failures](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Failures)で提供されています。 +## APIリファレンス {#api-reference} -## API Reference +### Podアノテーション -### Pod Annotation - -Specifying the profile a container will run with: +コンテナが実行するプロファイルを指定します。 - **key**: `container.apparmor.security.beta.kubernetes.io/` - Where `` matches the name of a container in the Pod. - A separate profile can be specified for each container in the Pod. -- **value**: a profile reference, described below + ここで、``はPod内のコンテナの名前を一致させます。Pod内の各コンテナごとに別々のプロファイルを指定できます。 +- **value**: 下で説明するプロファイルのリファレンス -### Profile Reference +### プロファイルのリファレンス -- `runtime/default`: Refers to the default runtime profile. - - Equivalent to not specifying a profile (without a PodSecurityPolicy default), except it still - requires AppArmor to be enabled. - - For Docker, this resolves to the - [`docker-default`](https://docs.docker.com/engine/security/apparmor/) profile for non-privileged - containers, and unconfined (no profile) for privileged containers. -- `localhost/`: Refers to a profile loaded on the node (localhost) by name. - - The possible profile names are detailed in the - [core policy reference](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Core_Policy_Reference#profile-names-and-attachment-specifications). -- `unconfined`: This effectively disables AppArmor on the container. +- `runtime/default`: デフォルトのランタイムプロファイルを指します。 + - (PodSecurityPolicyのデフォルトを設定せずに)プロファイルを指定しない場合と同等ですが、AppArmorを有効化する必要があります。 + - Dockerの場合、非特権コンテナでは[`docker-default`](https://docs.docker.com/engine/security/apparmor/)プロファイルが選択され、特権コンテナではunconfined(プロファイルなし)が選択されます。 +- `localhost/`: 名前で指定されたノード(localhost)に読み込まれたプロファイルを指します。 + - 利用できるプロファイル名の詳細は[core policy reference](https://gitlab.com/apparmor/apparmor/wikis/AppArmor_Core_Policy_Reference#profile-names-and-attachment-specifications)で説明されています。 +- `unconfined`: これは実質的にコンテナ上のAppArmorを無効化します。 -Any other profile reference format is invalid. +これ以外のプロファイルリファレンスはすべて無効です。 -### PodSecurityPolicy Annotations +### PodSecurityPolicyアノテーション -Specifying the default profile to apply to containers when none is provided: +noneが与えられたときにコンテナに適用するデフォルトのプロファイルは、以下のように指定します。 * **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` -* **value**: a profile reference, described above +* **value**: 上で説明したプロファイルのリファレンス -Specifying the list of profiles Pod containers is allowed to specify: +Podコンテナが指定することを許可するプロファイルのリストは、以下のように指定します。 * **key**: `apparmor.security.beta.kubernetes.io/allowedProfileNames` -* **value**: a comma-separated list of profile references (described above) - - Although an escaped comma is a legal character in a profile name, it cannot be explicitly - allowed here. - - +* **value**: カンマ区切りの上述のプロファイルリファレンスのリスト + - プロファイル名ではエスケープしたカンマは不正な文字ではありませんが、ここでは明示的に許可されません。 ## {{% heading "whatsnext" %}} - -Additional resources: +追加のリソースとしては以下のものがあります。 * [Quick guide to the AppArmor profile language](https://gitlab.com/apparmor/apparmor/wikis/QuickProfileLanguage) * [AppArmor core policy reference](https://gitlab.com/apparmor/apparmor/wikis/Policy_Layout) - - From 021dc964b3b7e5bb5a8c6a0ffa46c59c63bd343a Mon Sep 17 00:00:00 2001 From: wangyysde Date: Sun, 4 Apr 2021 22:41:02 +0800 Subject: [PATCH 043/218] Update content/zh/docs/concepts/cluster-administration/system-metrics.md Co-authored-by: Qiming Teng --- .../zh/docs/concepts/cluster-administration/system-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/concepts/cluster-administration/system-metrics.md b/content/zh/docs/concepts/cluster-administration/system-metrics.md index a7053da87d..e508e09e26 100644 --- a/content/zh/docs/concepts/cluster-administration/system-metrics.md +++ b/content/zh/docs/concepts/cluster-administration/system-metrics.md @@ -40,7 +40,7 @@ Examples of those components: ## Kubernetes 中组件的指标 在大多数情况下，可以通过HTTP访问组件endpoint的 `/metrics` 路径来获取组件的度量值。 -对于那些默认情况下不暴露的endpoint的组件，可以使用 `--bind-address` 标志启用。 +对于那些默认情况下不暴露端点的组件，可以使用 `--bind-address` 标志启用。 这些组件的示例： From 078331611735f537296d4c2b0b3259c7b93fddd5 Mon Sep 17 00:00:00 2001 From: wangyysde Date: Sun, 4 Apr 2021 22:41:17 +0800 Subject: [PATCH 044/218] Update content/zh/docs/concepts/cluster-administration/system-metrics.md Co-authored-by: Qiming Teng --- .../zh/docs/concepts/cluster-administration/system-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/concepts/cluster-administration/system-metrics.md b/content/zh/docs/concepts/cluster-administration/system-metrics.md index e508e09e26..033dabb95e 100644 --- a/content/zh/docs/concepts/cluster-administration/system-metrics.md +++ b/content/zh/docs/concepts/cluster-administration/system-metrics.md @@ -39,7 +39,7 @@ Examples of those components: --> ## Kubernetes 中组件的指标 -在大多数情况下，可以通过HTTP访问组件endpoint的 `/metrics` 路径来获取组件的度量值。 +在大多数情况下，可以通过 HTTP 访问组件的 `/metrics` 端点来获取组件的度量值。 对于那些默认情况下不暴露端点的组件，可以使用 `--bind-address` 标志启用。 这些组件的示例： From 21f41f2d18b889defe84a36c36e2e222d77250f5 Mon Sep 17 00:00:00 2001 From: Aris Cahyadi Risdianto Date: Sun, 4 Apr 2021 22:39:15 +0800 Subject: [PATCH 045/218] updating learning environment minikube. --- .../docs/setup/learning-environment/_index.md | 31 ++ .../setup/learning-environment/minikube.md | 514 ------------------ content/id/docs/tutorials/hello-minikube.md | 16 +- static/_redirects | 4 + 4 files changed, 43 insertions(+), 522 deletions(-) delete mode 100644 content/id/docs/setup/learning-environment/minikube.md diff --git a/content/id/docs/setup/learning-environment/_index.md b/content/id/docs/setup/learning-environment/_index.md index 4c116a9bc0..cdfe637e64 100644 --- a/content/id/docs/setup/learning-environment/_index.md +++ b/content/id/docs/setup/learning-environment/_index.md @@ -2,3 +2,34 @@ title: Lingkungan Pembelajaran weight: 20 --- + +## kind + +[`kind`](https://kind.sigs.k8s.io/docs/) memberikan kamu kemampuan untuk +menjalankan Kubernetes pada komputer lokal kamu. Perangkat ini membutuhkan +[Docker](https://docs.docker.com/get-docker/) yang sudah diinstal dan +terkonfigurasi. + +Halaman [Memulai Cepat](https://kind.sigs.k8s.io/docs/user/quick-start/) `kind` +memperlihatkan kepada kamu tentang apa yang perlu kamu lakukan agar `kind` dapat +berjalan dan bekerja. + +Melihat Memulai Cepat Kind + +## minikube + +Seperti halnya dengan `kind`, [`minikube`](https://minikube.sigs.k8s.io/) +merupakan perangkat yang memungkinkan kamu untuk menjalankan Kubernetes +secara lokal. `minikube` menjalankan sebuah klaster Kubernetes dengan +satu node saja dalam komputer pribadi (termasuk Windows, macOS dan Linux) +sehingga kamu dapat mencoba Kubernetes atau untuk pekerjaan pengembangan +sehari-hari. + +Kamu bisa mengikuti petunjuk resmi +[Memulai!](https://minikube.sigs.k8s.io/docs/start/) +`minikube` jika kamu ingin fokus agar perangkat ini terinstal. + +Lihat Panduan Memulai! Minikube + +Setelah kamu memiliki `minikube` yang bekerja, kamu bisa menggunakannya +untuk [menjalankan aplikasi contoh](/id/docs/tutorials/hello-minikube/). diff --git a/content/id/docs/setup/learning-environment/minikube.md b/content/id/docs/setup/learning-environment/minikube.md deleted file mode 100644 index 8729968738..0000000000 --- a/content/id/docs/setup/learning-environment/minikube.md +++ /dev/null @@ -1,514 +0,0 @@ ---- -title: Instalasi Kubernetes dengan Minikube -weight: 30 -content_type: concept ---- - - - -Minikube adalah alat yang memudahkan untuk menjalankan Kubernetes pada komputer lokal. Minikube menjalankan satu Node klaster Kubernetes di dalam _Virtual Machine_ (VM) pada laptop kamu untuk pengguna yang ingin mencoba Kubernetes atau mengembangkannya. - - - - -## Fitur Minikube - -Minikube mendukung fitur Kubernetes berikut: - -* DNS -* NodePort -* {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}} dan {< glossary_tooltip text="Secret" term_id="secret" >}} -* _Dashboard_ -* _Container runtime_: [Docker](https://www.docker.com/), [CRI-O](https://cri-o.io/), dan [containerd](https://github.com/containerd/containerd) -* {{< glossary_tooltip text="CNI" term_id="cni" >}} -* Ingress - -## Instalasi - -Lihat [Instalasi Minikube](/id/docs/tasks/tools/install-minikube/). - -## Memulai Cepat - -Demonstrasi singkat ini memandu kamu tentang bagaimana memulai, menggunakan dan menghapus Minikube secara lokal. Ikuti langkah berikut untuk memulai dan menjelajahi Minikube. - -1. Mulailah Minikube dan buatlah sebuah klaster: - - ```shell - minikube start - ``` - - Keluaran menyerupai: - - ``` - Starting local Kubernetes cluster... - Running pre-create checks... - Creating machine... - Starting local Kubernetes cluster... - ``` - Untuk informasi lebih lanjut mengenai bagaimana memulai klaster pada versi Kubernetes tertentu, VM atau Container _runtime_, lihatlah [Memulai klaster](#memulai-klaster). - -2. Kini kamu bisa berinteraksi dengan klaster kamu dengan kubectl. Untuk informasi lebih lanjut, lihatlah [Interaksi dengan klaster kamu](#interaksi-dengan-klaster-kamu). - - Mari kita buat Kubernetes Deployment menggunakan _image_ bernama `echoserver`, yaitu sebuah server HTTP sederhana dan buka layanan pada porta 8080 dengan menggunakan opsi `--port`. - - ```shell - kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10 - ``` - - Keluaran menyerupai: - - ``` - deployment.apps/hello-minikube created - ``` -3. Untuk mengakses Deployment `hello-minikube`, bukalah dia sebagai sebuah Service: - - ```shell - kubectl expose deployment hello-minikube --type=NodePort --port=8080 - ``` - - Opsi `--type=NodePort` menentukan tipe Service. - - Keluarannya menyerupai: - - ``` - service/hello-minikube exposed - ``` - -4. Pod `hello-minikube` saat ini telah dibuat namun kamu harus menunggu hingga Pod selesai dijalankan sebelum dapat mengaksesnya melalui Service yang telah dibuka. - - Cek apakah Pod sudah berjalan dan beroperasi: - - ```shell - kubectl get pod - ``` - - Jika keluaran menampilkan `STATUS` sebagai `ContainerCreating`, maka Pod sedang dalam proses pembuatan: - - ``` - NAME READY STATUS RESTARTS AGE - hello-minikube-3383150820-vctvh 0/1 ContainerCreating 0 3s - ``` - - Jika keluaran menampilkan `STATUS` sebagai `Running`, maka Pod sudah berjalan dan beroperasi: - - ``` - NAME READY STATUS RESTARTS AGE - hello-minikube-3383150820-vctvh 1/1 Running 0 13s - ``` - -5. Ambil URL Service yang telah dibuka untuk melihat Service secara detail: - - ```shell - minikube service hello-minikube --url - ``` - -6. Untuk melihat detail dari klaster lokal kamu, salin dan tempel URL yang kamu dapatkan dari keluaran pada peramban kamu. - - Keluarannya menyerupai: - - ``` - Hostname: hello-minikube-7c77b68cff-8wdzq - - Pod Information: - -no pod information available- - - Server values: - server_version=nginx: 1.13.3 - lua: 10008 - - Request Information: - client_address=172.17.0.1 - method=GET - real path=/ - query= - request_version=1.1 - request_scheme=http - request_uri=http://192.168.99.100:8080/ - - Request Headers: - accept=*/* - host=192.168.99.100:30674 - user-agent=curl/7.47.0 - - Request Body: - -no body in request- - ``` - - Jika kamu tidak lagi membutuhkan Service dan klaster, maka kamu bisa menghapusnya. - -7. Hapuslah Service `hello-minikube`: - - ```shell - kubectl delete services hello-minikube - ``` - - Keluarannya menyerupai: - - ``` - service "hello-minikube" deleted - ``` - -8. Hapuslah Deployment `hello-minikube`: - - ```shell - kubectl delete deployment hello-minikube - ``` - - Keluarannya menyerupai: - - ``` - deployment.extensions "hello-minikube" deleted - ``` - -9. Hentikanlah klaster Minikube lokal: - - ```shell - minikube stop - ``` - - Keluarannya menyerupai: - - ``` - Stopping "minikube"... - "minikube" stopped. - ``` - - Untuk informasi lebih lanjut, lihatlah [Menghentikan Klaster](#menghentikan-klaster). - -10. Hapuslah klaster Minikube lokal - - ```shell - minikube delete - ``` - Keluarannya menyerupai: - ``` - Deleting "minikube" ... - The "minikube" cluster has been deleted. - ``` - Untuk informasi lebih lanjut, lihat [Menghapus Klaster](#menghapus-klaster). - -## Mengelola Klaster - -### Memulai Klaster - -Perintah `minikube start` bisa digunakan untuk memulai klaster kamu. -Perintah ini membuat dan mengonfigurasi sebuah mesin virtual yang menjalankan klaster Kubernetes dengan satu Node. -Perintah ini juga mengonfigurasi instalasi [kubectl](/id/docs/user-guide/kubectl-overview/) untuk berkomunikasi dengan klaster ini. - -{{< note >}} -Jika kamu menggunakan proksi web, maka kamu harus meneruskan informasi berikut ini ke perintah `minikube start`: - -```shell -https_proxy= minikube start --docker-env http_proxy= --docker-env https_proxy= --docker-env no_proxy=192.168.99.0/24 -``` -Sayangnya, pengaturan dengan _environment variable_ saja tidak berguna. - -Minikube juga membuat konteks "minikube", dan menetapkannya sebagai bawaan di kubectl. -Untuk kembali menggunakan konteks ini, jalankan perintah: `kubectl config use-context minikube`. -{{< /note >}} - -#### Menentukan Versi Kubernetes - -Kamu bisa menentukan versi Kubernetes yang digunakan oleh Minikube dengan -menambahkan `--kubernetes-version` ke perintah `minikube start`. Sebagai -contoh, untuk menjalankan versi {{}}, kamu akan menjalankan perintah berikut: - -``` -minikube start --kubernetes-version {{< param "fullversion" >}} -``` -#### Menentukan _driver_ VM - -Kamu bisa mengubah _driver_ VM dengan menambahkan tanda `--driver=` pada `minikube start`. -Sebagai contoh: -```shell -minikube start --driver= -``` - -Minikube mendukung _driver_ berikut ini: -{{< note >}} -Lihat [_DRIVER_](https://minikube.sigs.k8s.io/docs/reference/drivers/) untuk detail tentang _driver_ yang didukung dan proses instalasi _plugin_. -{{< /note >}} - -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/drivers/docker/)) docker -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/drivers/virtualbox/)) virtualbox -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/drivers/podman/)) podman (TAHAP EXPERIMEN) -* vmwarefusion -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/reference/drivers/kvm2/)) kvm2 -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/reference/drivers/hyperkit/)) hyperkit -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/reference/drivers/hyperv/)) hyperv -Perlu diingat bahwa IP dibawah adalah dinamik dan bisa berubah. IP ini bisa diambil dengan `minikube ip`. -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/reference/drivers/vmware/)) vmware (_driver_ VMware terpadu) -* ([instalasi driver](https://minikube.sigs.k8s.io/docs/reference/drivers/parallels/)) parallels -* none (menjalankan komponen Kubernetes pada hos dan bukan pada mesin virtual. Kamu harus menjalankan Linux dan harus menginstal {{}}.) - -{{< caution >}} -Jika kamu menggunakan _driver_ `none`, beberapa komponen Kubernetes dijalankan sebagai Container istimewa yang memiliki efek samping di luar lingkungan Minikube. Efek samping tersebut berarti bahwa _driver_ `none` tidak direkomendasikan untuk komputer pribadi. -{{< /caution >}} - -#### Memulai klaster pada _runtime_ kontainer alternatif -Kamu bisa memulai Minikube pada _runtime_ kontainer berikut. -{{< tabs name="container_runtimes" >}} -{{% tab name="containerd" %}} -Untuk menggunakan [containerd](https://github.com/containerd/containerd) sebagai _runtime_ kontainer, jalankan: -```bash -minikube start \ - --network-plugin=cni \ - --enable-default-cni \ - --container-runtime=containerd \ - --bootstrapper=kubeadm -``` - -Atau kamu bisa menggunakan versi yang diperpanjang: - -```bash -minikube start \ - --network-plugin=cni \ - --enable-default-cni \ - --extra-config=kubelet.container-runtime=remote \ - --extra-config=kubelet.container-runtime-endpoint=unix:///run/containerd/containerd.sock \ - --extra-config=kubelet.image-service-endpoint=unix:///run/containerd/containerd.sock \ - --bootstrapper=kubeadm -``` -{{% /tab %}} -{{% tab name="CRI-O" %}} -Untuk menggunakan [CRI-O](https://cri-o.io/) sebagain _runtime_ kontainer, jalankan: -```bash -minikube start \ - --network-plugin=cni \ - --enable-default-cni \ - --container-runtime=cri-o \ - --bootstrapper=kubeadm -``` -Atau kamu bisa menggunakan versi yang diperpanjang: - -```bash -minikube start \ - --network-plugin=cni \ - --enable-default-cni \ - --extra-config=kubelet.container-runtime=remote \ - --extra-config=kubelet.container-runtime-endpoint=/var/run/crio.sock \ - --extra-config=kubelet.image-service-endpoint=/var/run/crio.sock \ - --bootstrapper=kubeadm -``` -{{% /tab %}} -{{< /tabs >}} - -#### Menggunakan _image_ lokal degan menggunakan kembali _daemon_ Docker - -Saat menggunakan sebuah VM untuk Kubernetes, akan lebih baik jika _daemon_ Docker bawaan Minikube digunakan kembali. Menggunakan kembali _daemon_ bawaan membuat kamu tidak perlu membangun registri Docker pada mesin hos kamu dan mengunggah _image_ ke dalamnya. Namun, kamu dapat membangun di dalam _daemon_ Docker yang sama dengan Minikube, yang tentunya dapat mempercepat percobaan lokal. - -{{< note >}} -Pastikan untuk memberi _tag_ pada Docker _image_ kamu dengan sesuatu selain `latest` dan gunakan _tag_ tersebut untuk menarik _image_. Karena `:latest` adalah bawaan, dengan kebijakan penarikan _image_ bawaan, yaitu `Always`, kesalahan penarikan _image_ (`ErrImagePull`) akhirnya dapat terjadi jika kamu tidak memiliki _image_ Docker di register Docker bawaan (biasanya DockerHub). -{{< /note >}} - -Untuk bekerja dengan _daemon_ Docker pada mesin Mac/Linux, jalankan baris terakhir dari `minikube docker-env`. - -Kamu sekarang dapat menggunakan Docker di terminal mesin Mac/Linux kamu untuk berkomunikasi dengan _daemon_ Docker di dalam VM Minikube: - -```shell -docker ps -``` - -{{< note >}} -Pada Centos 7, Docker bisa memberikan kesalahan berikut: - -``` -Could not read CA certificate "/etc/docker/ca.pem": open /etc/docker/ca.pem: no such file or directory -``` - -Kamu bisa memperbaikinya dengan memperbaharui /etc/sysconfig/docker untuk memastikan bahwa lingkungan Minikube dikenali: - -```shell -< DOCKER_CERT_PATH=/etc/docker ---- -> if [ -z "${DOCKER_CERT_PATH}" ]; then -> DOCKER_CERT_PATH=/etc/docker -> fi -``` -{{< /note >}} - -### Mengonfigurasi Kubernetes - -Minikube memiliki sebuah fitur "pengonfigurasi" yang memperbolehkan pengguna untuk mengkonfigurasi komponen Kubernetes dengan sembarang nilai. -Untuk menggunakan fitur ini, kamu bisa menggunakan _flag_ `--extra-config` pada perintah `minikube start`. - -_Flag_ ini berulang, jadi kamu bisa menggunakannya beberapa kali dengan beberapa nilai yang berbeda untuk mengatur beberapa opsi. - -_Flag_ ini menerima sebuah _string_ dalam format `component.key=value`, di mana `component` adalah salah satu _string_ dari list di bawah, `key` adalah nilai dari _struct_ configurasi dan `value` adalah nilai yang digunakan. - -Kunci yang valid bisa ditemukan dengan memeriksa dokumentasi `componentconfigs` Kubernetes untuk setiap komponen. -Berikut adalah dokumentasi untuk setiap konfigurasi yang didukung: - -* [kubelet](https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/config#KubeletConfiguration) -* [apiserver](https://godoc.org/k8s.io/kubernetes/cmd/kube-apiserver/app/options#ServerRunOptions) -* [proxy](https://godoc.org/k8s.io/kubernetes/pkg/proxy/apis/config#KubeProxyConfiguration) -* [controller-manager](https://godoc.org/k8s.io/kubernetes/pkg/controller/apis/config#KubeControllerManagerConfiguration) -* [etcd](https://godoc.org/github.com/coreos/etcd/etcdserver#ServerConfig) -* [scheduler](https://godoc.org/k8s.io/kubernetes/pkg/scheduler/apis/config#KubeSchedulerConfiguration) - -#### Contoh - -Untuk mengubah pengaturan `MaxPods` menjadi 5 pada Kubelet, gunakan _flag_ ini: `--extra-config=kubelet.MaxPods=5`. - -Fitur ini juga mendukung _struct_ yang berulang. Untuk mengubah pengaturan `LeaderElection.LeaderElect` menjadi `true` pada penjadwal, gunakan _flag_: `--extra-config=scheduler.LeaderElection.LeaderElect=true`. - -Untuk mengatur `AuthorizationMode` pada `apiserver` menjadi `RBAC`, kamu bisa menggunakan: `--extra-config=apiserver.authorization-mode=RBAC`. - -### Menghentikan klaster -Perintah `minikube stop` bisa digunakan untuk menghentikan klaster kamu. -Perintah ini menghentikan mesin virtual Minikube, tapi mempertahankan semua status dan data klaster. -Memulai klaster lagi akan mengembalikannya ke keadaan sebelumnya. - -### Menghapus klaster -Perintah `minikube delete` bisa digunakan untuk menghapus klaster kamu. -Perintah ini menghentikan dan menghapus mesin virtual Minikube. Tidak ada data atau _state_ yang dipertahankan. - -### Memperbaharui Minikube -Jika kamu menggunakan MacOS dan [Brew Package Manager](https://brew.sh/) sudah terpasang, jalankan: - -```shell -brew update -brew upgrade minikube -``` - -## Interaksi dengan Klaster Kamu - -### Kubectl - -Perintah `minikube start` membuat sebuah [konteks kubectl](/id/docs/reference/generated/kubectl/kubectl-commands#-em-set-context-em-) yang disebut "minikube". -Konteks ini menyimpan pengaturan untuk berkomunikasi dengan klaster Minikube kamu. - -Minikube menetapkan konteks ini sebagai bawaan secara otomatis, tetapi jika kamu ingin mengubah kembali ke konteks tersebut di kemudian hari, gunakan: - -`kubectl config use-context minikube` - -Atau berikan konteks untuk setiap perintah seperti ini: - -`kubectl get pods --context=minikube` - -### Dashboard - -Untuk mengakses [Kubernetes Dashboard](/docs/tasks/access-application-cluster/web-ui-dashboard/), gunakan perintah ini pada terminal setelah memulai Minikube untuk mendapatkan alamatnya: - -```shell -minikube dashboard -``` - -### Service - -Untuk mengakses Service yang dibuka via NodePort, jalankan perintah ini pada terminal setelah memulai Minikube untuk mendapatkan alamat: - -```shell -minikube service [-n NAMESPACE] [--url] NAME -``` - -## Jaringan - -Mesin virtual Minikube dibuka ke sistem hos melalui alamat IP _host-only_ , yang bisa didapatkan dengan perintah `minikube ip`. -Seluruh Service dengan jenis `NodePort` bisa diakses melalui alamat IP pada NodePort. - -Untuk mementukan NodePort pada Service kamu, kamu bisa menggunakan perintah `kubectl` sebagai berikut: - -`kubectl get service $SERVICE --output='jsonpath="{.spec.ports[0].nodePort}"'` - -## PersistentVolume - -Minikube mendukung [PersistentVolume](/id/docs/concepts/storage/persistent-volumes/) dengan jenis `hostPath`. -PersistenVolume ini dipetakan ke direktori di dalam mesin virtual Minikube. - -Mesin virtual Minikube melakukan _booting_ ke tmpfs, sehingga sebagian besar direktori tidak akan bertahan setelah di _reboot_ (`minikube stop`). - -Namun, Minikube diatur untuk mempertahankan berkas yang tersimpan didalam direktori hos berikut: - -* `/data` -* `/var/lib/minikube` -* `/var/lib/docker` - -Ini adalah contoh pengaturan PersistentVolume untuk mempertahankan data di dalam direktori `/data`: - -```yaml -apiVersion: v1 -kind: PersistentVolume -metadata: - name: pv0001 -spec: - accessModes: - - ReadWriteOnce - capacity: - storage: 5Gi - hostPath: - path: /data/pv0001/ -``` - -## Folder hos yang di _mount_ -Beberapa _driver_ akan memasang folder _hos_ dalam VM sehingga kamu dapat dengan mudah berbagi berkas antara VM dan hos. Saat ini, hal tersebut tidak dapat dikonfigurasi dan berbeda untuk setiap _driver_ dan sistem operasi yang kamu gunakan. - -{{< note >}} -Berbagi folder hos belum diimplementasikan pada _driver_ KVM. -{{< /note >}} - -| Driver | OS | HostFolder | VM | -| --- | --- | --- | --- | -| VirtualBox | Linux | /home | /hosthome | -| VirtualBox | macOS | /Users | /Users | -| VirtualBox | Windows | C://Users | /c/Users | -| VMware Fusion | macOS | /Users | /mnt/hgfs/Users | -| Xhyve | macOS | /Users | /Users | - -## Registri Container Pribadi - -Untuk mengakses registri Container pribadi, ikuti langkah berikut pada [halaman ini](/id/docs/concepts/containers/images/). - -Kami merekomendasi penggunaan `ImagePullSecrets`, tetapi jika kamu ingin mengonfigurasi akses pada virtual mesin Minikube, kamu bisa menempatkan `.dockercfg` pada direktori `/home/docker` atau `config.json` dalam direktori `/home/docker/.docker`. - -## Tambahan (_Add-on_) - -Supaya Minikube memulai atau memulai kembali kustom tambahan dengan benar, -tempatkan tambahan yang ingin kamu jalankan di dalam direktori `~/.minikube/addons`. -Tambahan dalam folder akan dipindahkan ke virtual mesin Minikube dan dijalankan setiap kali Minikube -dimulai atau dimulai ulang. - -## Menggunakan Minikube dengan Proksi HTTP - -Minikube membuat sebuah mesin virtual yang memasukkan Kubernetes dan _daemon_ Docker. -Ketika Kubernetes berusaha untuk menjadwalkan Container dengan Docker, _daemon_ Docker mungkin membutuhkan -akses jaringan eksternal untuk menarik Container. - -Jika kamu berada di belakang _proxy_ HTTP, kamu mungkin perlu menyediakan Docker dengan pengaturan proksi. -Untuk melakukan ini, berikan _environment variable_ yang dibutuhkan sebagai _flag_ pada saat `minikube start`. - -Contoh: - -```shell -minikube start --docker-env http_proxy=http://$YOURPROXY:PORT \ - --docker-env https_proxy=https://$YOURPROXY:PORT -``` - -Jika alamat mesin virtual kamu adalah 192.168.99.100, maka ada kemungkinan pengaturan proksi kamu akan mencegah `kubectl` untuk mencapainya. -Untuk melewatkan konfigurasi _proxy_ untuk alamat IP ini, kamu harus memodifikasi pengaturan _no_proxy` kamu. Kamu bisa melakukannya dengan: - -```shell -export no_proxy=$no_proxy,$(minikube ip) -``` - -## Masalah yang Diketahui - -Fitur yang memerlukan banyak Node tidak akan berfungsi dalam Minikube. - -## Desain - -Minikube menggunakan [libmachine](https://github.com/docker/machine/tree/master/libmachine) untuk menyediakan mesin virtual, dan [kubeadm](https://github.com/kubernetes/kubeadm) untuk menyediakan klaster Kubernetes. - -Untuk info lebih lanjut tentang Minikube, lihat [proposal](https://git.k8s.io/community/contributors/design-proposals/cluster-lifecycle/local-cluster-ux.md). - -## Tautan Tambahan - -* **Tujuan and Non-Tujuan**: Untuk tujuan dan non-tujuan dari projek Minikube, lihat [roadmap](https://minikube.sigs.k8s.io/docs/contrib/roadmap/). -* **Petunjuk Pengembangan**: Lihat [Berkontribusi](https://minikube.sigs.k8s.io/docs/contrib/) untuk ikhtisar bagaimana cara mengirimkan _pull request_. -* **Membangun Minikube**: Untuk instruksi bagaimana membangun atau mengetes Minikube dari sumber kode, lihat [petunjuk membangun](https://minikube.sigs.k8s.io/docs/contrib/building/). -* **Menambahkan Dependensi Baru**: Untuk instruksi bagaimana menambahkan dependensi baru ke Minikube, lihat [petunjuk penambahan dependensi](https://minikube.sigs.k8s.io/docs/contrib/drivers/). -* **Menambahkan Addon Baru**: Untuk instruksi bagaimana menambahkan tambahan baru untuk Minikube, lihat [petunjuk menambahkan addon baru](https://minikube.sigs.k8s.io/docs/contrib/addons/). -* **MicroK8s**: Pengguna Linux yang ingin menghindari penggunaan mesin virtual, bisa mempertimbangkan [MicroK8s](https://microk8s.io/) sebagai alternatif. - -## Komunitas - -Kontribusi, pertanyaan, dan komentar sangat diharapkan! Pengembang Minikube berkumpul dalam [Slack](https://kubernetes.slack.com) di _channel_ #minikube (dapatkan undangan [di sini](http://slack.kubernetes.io/)). Kami juga memiliki [milis kubernetes-dev Google Groups](https://groups.google.com/forum/#!forum/kubernetes-dev). Jika kamu memposting sesuatu, awali subjek kamu dengan "minikube: ". diff --git a/content/id/docs/tutorials/hello-minikube.md b/content/id/docs/tutorials/hello-minikube.md index faba283d89..398c5a3a3f 100644 --- a/content/id/docs/tutorials/hello-minikube.md +++ b/content/id/docs/tutorials/hello-minikube.md @@ -15,11 +15,11 @@ card: -Tutorial ini menunjukkan bagaimana caranya menjalankan aplikasi sederhana Node.js Halo Dunia di Kubernetes, dengan [Minikube](/docs/getting-started-guides/minikube) dan Katacoda. +Tutorial ini menunjukkan bagaimana caranya menjalankan aplikasi sederhana Node.js Halo Dunia di Kubernetes, dengan [`minikube`](/docs/getting-started-guides/minikube) dan Katacoda. Katacoda menyediakan environment Kubernetes secara gratis di dalam browser. {{< note >}} -Kamupun bisa mengikuti tutorial ini kalau sudah instalasi [Minikube di lokal](/id/docs/tasks/tools/install-minikube/) kamu. +Kamupun bisa mengikuti tutorial ini kalau sudah instalasi minikube di lokal. Silakan lihat [memulai `minikube`](https://minikube.sigs.k8s.io/docs/start/) untuk instruksi instalasi. {{< /note >}} @@ -27,7 +27,7 @@ Kamupun bisa mengikuti tutorial ini kalau sudah instalasi [Minikube di lokal](/i ## {{% heading "objectives" %}} -* Deploy aplikasi halo dunia pada Minikube. +* Deploy aplikasi halo dunia pada minikube. * Jalankan aplikasinya. * Melihat log aplikasi. @@ -54,7 +54,7 @@ Untuk info lebih lanjut tentang perintah `docker build`, baca [dokumentasi Docke {{< kat-button >}} - {{< note >}}Kalau kamu memilih instalasi Minikube secara lokal, jalankan `minikube start`.{{< /note >}} + {{< note >}}Kalau kamu memilih instalasi minikube secara lokal, jalankan `minikube start`.{{< /note >}} 2. Buka dasbor Kubernetes di dalam browser: @@ -147,7 +147,7 @@ Supaya Kontainer `hello-node` bisa diakses dari luar jaringan virtual Kubernetes ``` Untuk penyedia cloud yang memiliki load balancer, sebuah alamat IP eksternal akan disediakan untuk mengakses Servis tersebut. - Pada Minikube, tipe `LoadBalancer` membuat Servis tersebut dapat diakses melalui perintah `minikube service`. + Pada minikube, tipe `LoadBalancer` membuat Servis tersebut dapat diakses melalui perintah `minikube service`. 3. Jalankan perintah berikut: @@ -163,7 +163,7 @@ Supaya Kontainer `hello-node` bisa diakses dari luar jaringan virtual Kubernetes ## Aktifkan addons -Minikube punya beberapa addons yang bisa diaktifkan, dinon-aktifkan, maupun dibuka di dalam environment Kubernetes lokal. +Perangkat minikube meliputi sekumpulan {{< glossary_tooltip text="addons" term_id="addons" >}} bawaan yang bisa diaktifkan, dinonaktifkan, maupun dibuka di dalam environment Kubernetes lokal. 1. Daftar addons yang ada saat ini: @@ -249,13 +249,13 @@ kubectl delete service hello-node kubectl delete deployment hello-node ``` -Kamu juga boleh mematikan mesin virtual (VM) untuk Minikube: +Kamu juga boleh mematikan mesin virtual atau _virtual machine_ (VM) untuk minikube: ```shell minikube stop ``` -Kamu juga boleh menghapus Minikube VM: +Kamu juga boleh menghapus minikube VM: ```shell minikube delete diff --git a/static/_redirects b/static/_redirects index db9f04df02..fdc9985dea 100644 --- a/static/_redirects +++ b/static/_redirects @@ -475,10 +475,14 @@ /docs/setup/version-skew-policy/ /docs/setup/release/version-skew-policy/ 301 /docs/setup/minikube/ /docs/tasks/tools/ 302 +/id/docs/setup/minikube/ /id/docs/tasks/tools/ 302 /docs/setup/learning-environment/ /docs/tasks/tools/ 302! +/id/docs/setup/learning-environment/ /id/docs/tasks/tools/ 302! /zh/docs/setup/learning-environment/ /zh/docs/tasks/tools/ 302! /docs/setup/learning-environment/kind/ /docs/tasks/tools/ 302 +/id/docs/setup/learning-environment/kind/ /id/docs/tasks/tools/ 302 /docs/setup/learning-environment/minikube/ /docs/tasks/tools/ 302 +/id/docs/setup/learning-environment/minikube/ /id/docs/tasks/tools/ 302 /docs/setup/cri/ /docs/setup/production-environment/container-runtimes/ 301 /docs/setup/independent/install-kubeadm/ /docs/setup/production-environment/tools/kubeadm/install-kubeadm/ 301 /ja/docs/setup/independent/install-kubeadm/ /ja/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ 301 From 88275ce3fe3d56912a5382c6f2c5ff0cd847b98f Mon Sep 17 00:00:00 2001 From: TAKAHASHI Shuuji Date: Tue, 6 Apr 2021 18:17:05 +0900 Subject: [PATCH 046/218] Fix wrong translations --- content/ja/docs/tutorials/clusters/apparmor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/ja/docs/tutorials/clusters/apparmor.md b/content/ja/docs/tutorials/clusters/apparmor.md index 6f5bd08709..832e684f44 100644 --- a/content/ja/docs/tutorials/clusters/apparmor.md +++ b/content/ja/docs/tutorials/clusters/apparmor.md @@ -302,7 +302,7 @@ apparmor.security.beta.kubernetes.io/defaultProfileName: apparmor.security.beta.kubernetes.io/allowedProfileNames: [,others...] ``` -defaultProfileNameオプションには、noneが指定された場合にコンテナにデフォルトで適用されるプロファイルを指定します。allowedProfileNamesオプションには、Podコンテナの実行が許可されるプロファイルのリストを指定します。両方のオプションが指定された場合、デフォルトは許可されなければいけません。プロファイルはコンテナ上で同じフォーマットで指定されます。完全な仕様については、[APIリファレンス](#api-reference)を参照してください。 +defaultProfileNameオプションには、何も指定されなかった場合にコンテナにデフォルトで適用されるプロファイルを指定します。allowedProfileNamesオプションには、Podコンテナの実行が許可されるプロファイルのリストを指定します。両方のオプションが指定された場合、デフォルトは許可されなければいけません。プロファイルはコンテナ上で同じフォーマットで指定されます。完全な仕様については、[APIリファレンス](#api-reference)を参照してください。 ### AppArmorの無効化 @@ -356,7 +356,7 @@ AppArmorに関する問題をデバッグするには、システムログをチ ### PodSecurityPolicyアノテーション -noneが与えられたときにコンテナに適用するデフォルトのプロファイルは、以下のように指定します。 +何も指定されな買った場合にコンテナに適用するデフォルトのプロファイルは、以下のように指定します。 * **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` * **value**: 上で説明したプロファイルのリファレンス From eab8f5361c982068731d8cf341ad9c59b184017d Mon Sep 17 00:00:00 2001 From: TAKAHASHI Shuuji Date: Tue, 6 Apr 2021 18:19:45 +0900 Subject: [PATCH 047/218] Fix a typo --- content/ja/docs/tutorials/clusters/apparmor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/tutorials/clusters/apparmor.md b/content/ja/docs/tutorials/clusters/apparmor.md index 832e684f44..5ee0201d71 100644 --- a/content/ja/docs/tutorials/clusters/apparmor.md +++ b/content/ja/docs/tutorials/clusters/apparmor.md @@ -356,7 +356,7 @@ AppArmorに関する問題をデバッグするには、システムログをチ ### PodSecurityPolicyアノテーション -何も指定されな買った場合にコンテナに適用するデフォルトのプロファイルは、以下のように指定します。 +何も指定されなかった場合にコンテナに適用するデフォルトのプロファイルは、以下のように指定します。 * **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` * **value**: 上で説明したプロファイルのリファレンス From 668e47312f9ddf5c1d3962654acdf7d61d7b3d9a Mon Sep 17 00:00:00 2001 From: Rael Garcia Date: Tue, 6 Apr 2021 12:53:59 +0200 Subject: [PATCH 048/218] clean: update spanish owners file Updates OWNERs file to match `kubernets/org` file updated at https://github.com/kubernetes/org/pull/2522. --- OWNERS_ALIASES | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES index c31b04d5b0..18e11bcfe9 100644 --- a/OWNERS_ALIASES +++ b/OWNERS_ALIASES @@ -50,11 +50,9 @@ aliases: - zparnold sig-docs-es-owners: # Admins for Spanish content - raelga - - alexbrand + - electrocucaracha sig-docs-es-reviews: # PR reviews for Spanish content - raelga - - alexbrand - # glo-pena - electrocucaracha sig-docs-fr-owners: # Admins for French content - remyleone From 81a1865fd58b1b692c88465397f3f7769767e699 Mon Sep 17 00:00:00 2001 From: Aris Cahyadi Risdianto Date: Wed, 7 Apr 2021 00:33:37 +0800 Subject: [PATCH 049/218] moving "assign-pod" and "taint & toleration" into scheduling-eviction. --- content/id/docs/concepts/architecture/nodes.md | 2 +- .../docs/concepts/containers/runtime-class.md | 2 +- .../overview/working-with-objects/labels.md | 2 +- .../assign-pod-node.md | 4 ++-- .../kube-scheduler.md | 0 .../taint-and-toleration.md | 0 .../concepts/services-networking/service.md | 2 +- .../docs/concepts/storage/storage-classes.md | 6 +++--- .../workloads/controllers/daemonset.md | 4 ++-- content/id/docs/reference/glossary/taint.md | 18 ++++++++++++++++++ .../id/docs/reference/glossary/toleration.md | 17 +++++++++++++++++ .../tasks/administer-cluster/sysctl-cluster.md | 2 +- static/_redirects | 2 ++ 13 files changed, 49 insertions(+), 12 deletions(-) rename content/id/docs/concepts/{configuration => scheduling-eviction}/assign-pod-node.md (98%) rename content/id/docs/concepts/{scheduling => scheduling-eviction}/kube-scheduler.md (100%) rename content/id/docs/concepts/{configuration => scheduling-eviction}/taint-and-toleration.md (100%) create mode 100644 content/id/docs/reference/glossary/taint.md create mode 100644 content/id/docs/reference/glossary/toleration.md diff --git a/content/id/docs/concepts/architecture/nodes.md b/content/id/docs/concepts/architecture/nodes.md index a4cae113e6..685d54dddd 100644 --- a/content/id/docs/concepts/architecture/nodes.md +++ b/content/id/docs/concepts/architecture/nodes.md @@ -166,7 +166,7 @@ Pada kasus ini, kontroler node berasumsi ada masalah pada jaringan master, dan m Mulai dari Kubernetes 1.6, kontroler node juga bertanggung jawab untuk melakukan eviction pada pod-pod yang berjalan di atas node dengan taints `NoExecute`, ketika pod-pod tersebut sudah tidak lagi tolerate terhadap taints. Sebagai tambahan, hal ini di-nonaktifkan secara default pada fitur alpha, kontroler node bertanggung jawab untuk menambahkan taints yang berhubungan dengan masalah pada node, seperti terputus atau `NotReady`. -Lihat [dokumentasi ini](/id/docs/concepts/configuration/taint-and-toleration/) untuk bahasan detail tentang taints `NoExecute` dan fitur alpha. +Lihat [dokumentasi ini](/id/docs/concepts/scheduling-eviction/taint-and-toleration/) untuk bahasan detail tentang taints `NoExecute` dan fitur alpha. Mulai dari versi 1.8, kontroler node bisa diatur untuk bertanggung jawab pada pembuatan taints yang merepresentasikan node condition. Ini merupakan fitur alpha untuk versi 1.8. diff --git a/content/id/docs/concepts/containers/runtime-class.md b/content/id/docs/concepts/containers/runtime-class.md index 73252a03e4..539fbb7038 100644 --- a/content/id/docs/concepts/containers/runtime-class.md +++ b/content/id/docs/concepts/containers/runtime-class.md @@ -45,7 +45,7 @@ soal bagaimana melakukan konfigurasi untuk implementasi CRI yang kamu miliki. Untuk saat ini, RuntimeClass berasumsi bahwa semua _node_ di dalam klaster punya konfigurasi yang sama (homogen). Jika ada _node_ yang punya konfigurasi berbeda dari yang lain (heterogen), maka perbedaan ini harus diatur secara independen di luar RuntimeClass -melalui fitur _scheduling_ (lihat [Menempatkan Pod pada Node](/id/docs/concepts/configuration/assign-pod-node/)). +melalui fitur _scheduling_ (lihat [Menempatkan Pod pada Node](/id/docs/concepts/scheduling-eviction/assign-pod-node/)). {{< /note >}} Seluruh konfigurasi memiliki nama `handler` yang terkait, dijadikan referensi oleh RuntimeClass. diff --git a/content/id/docs/concepts/overview/working-with-objects/labels.md b/content/id/docs/concepts/overview/working-with-objects/labels.md index 306edc0bfb..7b4b125062 100644 --- a/content/id/docs/concepts/overview/working-with-objects/labels.md +++ b/content/id/docs/concepts/overview/working-with-objects/labels.md @@ -220,6 +220,6 @@ selector: #### Memilih kumpulan Node Salah satu contoh penggunaan pemilihan dengan menggunakan label yaitu untuk membatasi suatu kumpulan Node tertentu yang dapat digunakan oleh Pod. -Lihat dokumentasi pada [pemilihan Node](/id/docs/concepts/configuration/assign-pod-node/) untuk informasi lebih lanjut. +Lihat dokumentasi pada [pemilihan Node](/id/docs/concepts/scheduling-eviction/assign-pod-node/) untuk informasi lebih lanjut. diff --git a/content/id/docs/concepts/configuration/assign-pod-node.md b/content/id/docs/concepts/scheduling-eviction/assign-pod-node.md similarity index 98% rename from content/id/docs/concepts/configuration/assign-pod-node.md rename to content/id/docs/concepts/scheduling-eviction/assign-pod-node.md index ee9e8bf2f4..f8654f4276 100644 --- a/content/id/docs/concepts/configuration/assign-pod-node.md +++ b/content/id/docs/concepts/scheduling-eviction/assign-pod-node.md @@ -114,7 +114,7 @@ Berikut ini contoh dari pod yang menggunakan afinitas node: Aturan afinitas node tersebut menyatakan pod hanya bisa ditugaskan pada node dengan label yang memiliki kunci `kubernetes.io/e2e-az-name` dan bernilai `e2e-az1` atau `e2e-az2`. Selain itu, dari semua node yang memenuhi kriteria tersebut, mode dengan label dengan kunci `another-node-label-key` and bernilai `another-node-label-value` harus lebih diutamakan. -Kamu dapat meilhat operator `In` digunakan dalam contoh berikut. Sitaksis afinitas node yang baru mendukung operator-operator berikut: `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt`. Kamu dapat menggunakan `NotIn` dan `DoesNotExist` untuk mewujudkan perilaku node anti-afinitas, atau menggunakan [node taints](/id/docs/concepts/configuration/taint-and-toleration/) untuk menolak pod dari node tertentu. +Kamu dapat meilhat operator `In` digunakan dalam contoh berikut. Sitaksis afinitas node yang baru mendukung operator-operator berikut: `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt`. Kamu dapat menggunakan `NotIn` dan `DoesNotExist` untuk mewujudkan perilaku node anti-afinitas, atau menggunakan [node taints](/id/docs/concepts/scheduling-eviction/taint-and-toleration/) untuk menolak pod dari node tertentu. Jika kamu menyatakan `nodeSelector` dan `nodeAffinity`. *keduanya* harus dipenuhi agar pod dapat dijadwalkan pada node kandidat. @@ -284,7 +284,7 @@ Lihat [tutorial ZooKeeper](/docs/tutorials/stateful-application/zookeeper/#toler Untuk informasi lebih lanjut tentang afinitas/anti-afinitas antar pod, lihat [design doc](https://git.k8s.io/community/contributors/design-proposals/scheduling/podaffinity.md). -Kamu juga dapat mengecek [Taints](/id/docs/concepts/configuration/taint-and-toleration/), yang memungkinkan sebuah *node* untuk *menolak* sekumpulan pod. +Kamu juga dapat mengecek [Taints](/id/docs/concepts/scheduling-eviction/taint-and-toleration/), yang memungkinkan sebuah *node* untuk *menolak* sekumpulan pod. ## nodeName diff --git a/content/id/docs/concepts/scheduling/kube-scheduler.md b/content/id/docs/concepts/scheduling-eviction/kube-scheduler.md similarity index 100% rename from content/id/docs/concepts/scheduling/kube-scheduler.md rename to content/id/docs/concepts/scheduling-eviction/kube-scheduler.md diff --git a/content/id/docs/concepts/configuration/taint-and-toleration.md b/content/id/docs/concepts/scheduling-eviction/taint-and-toleration.md similarity index 100% rename from content/id/docs/concepts/configuration/taint-and-toleration.md rename to content/id/docs/concepts/scheduling-eviction/taint-and-toleration.md diff --git a/content/id/docs/concepts/services-networking/service.md b/content/id/docs/concepts/services-networking/service.md index 5136a00415..3c85c5abbf 100644 --- a/content/id/docs/concepts/services-networking/service.md +++ b/content/id/docs/concepts/services-networking/service.md @@ -745,7 +745,7 @@ dan tidak akan menerima trafik apa pun. Untuk menghasilkan distribusi trafik yang merata, kamu dapat menggunakan _DaemonSet_ atau melakukan spesifikasi -[pod anti-affinity](/id/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature) +[pod anti-affinity](/id/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature) agar `Pod` tidak di-_assign_ ke _node_ yang sama. NLB juga dapat digunakan dengan anotasi [internal load balancer](/id/docs/concepts/services-networking/service/#internal-load-balancer). diff --git a/content/id/docs/concepts/storage/storage-classes.md b/content/id/docs/concepts/storage/storage-classes.md index 2897399e80..c5fc71a8de 100644 --- a/content/id/docs/concepts/storage/storage-classes.md +++ b/content/id/docs/concepts/storage/storage-classes.md @@ -149,10 +149,10 @@ PersistentVolumeClaim dibuat. PersistentVolume akan dipilih atau di-_provisionin sesuai dengan topologi yang dispesifikasikan oleh limitasi yang diberikan oleh mekanisme _scheduling_ Pod. Hal ini termasuk, tetapi tidak hanya terbatas pada, [persyaratan sumber daya](/id/docs/concepts/configuration/manage-compute-resources-container), -[_node selector_](/id/docs/concepts/configuration/assign-pod-node/#nodeselector), +[_node selector_](/id/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector), [afinitas dan -anti-afinitas Pod](/id/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity), -serta [_taint_ dan _toleration_](/id/docs/concepts/configuration/taint-and-toleration). +anti-afinitas Pod](/id/docs/concepts/scheduling-evictionassign-pod-node/#affinity-and-anti-affinity), +serta [_taint_ dan _toleration_](/id/docs/concepts/scheduling-eviction/taint-and-toleration). Beberapa _plugin_ di bawah ini mendukung `WaitForFirstConsumer` dengan _provisioning_ dinamis: diff --git a/content/id/docs/concepts/workloads/controllers/daemonset.md b/content/id/docs/concepts/workloads/controllers/daemonset.md index 0b1c0e71e9..905a0a193a 100644 --- a/content/id/docs/concepts/workloads/controllers/daemonset.md +++ b/content/id/docs/concepts/workloads/controllers/daemonset.md @@ -97,8 +97,8 @@ membuat Pod dengan nilai yang berbeda di sebuah Node untuk _testing_. Jika kamu menspesifikasikan `.spec.template.spec.nodeSelector`, maka _controller_ DaemonSet akan membuat Pod pada Node yang cocok dengan [selektor -Node](/id/docs/concepts/configuration/assign-pod-node/). Demikian juga, jika kamu menspesifikasikan `.spec.template.spec.affinity`, -maka _controller_ DaemonSet akan membuat Pod pada Node yang cocok dengan [Node affinity](/id/docs/concepts/configuration/assign-pod-node/). +Node](/id/docs/concepts/scheduling-eviction/assign-pod-node/). Demikian juga, jika kamu menspesifikasikan `.spec.template.spec.affinity`, +maka _controller_ DaemonSet akan membuat Pod pada Node yang cocok dengan [Node affinity](/id/docs/concepts/scheduling-eviction/assign-pod-node/). Jika kamu tidak menspesifikasikan sama sekali, maka _controller_ DaemonSet akan membuat Pod pada semua Node. diff --git a/content/id/docs/reference/glossary/taint.md b/content/id/docs/reference/glossary/taint.md new file mode 100644 index 0000000000..1545a025fd --- /dev/null +++ b/content/id/docs/reference/glossary/taint.md @@ -0,0 +1,18 @@ +--- +title: Taint +id: taint +date: 2019-01-11 +full_link: /id/docs/concepts/scheduling-eviction/taint-and-toleration/ +short_description: > + Objek inti yang terdiri dari tiga properti yang diperlukan: _key_(kunci), _value_(nilai), dan _effect_(efek). Taint mencegah penjadwalan Pod pada Node atau grup Node. + +aka: +tags: +- core-object +- fundamental +--- + Objek inti yang terdiri dari tiga properti yang diperlukan: _key_(kunci), _value_(nilai), dan _effect_(efek). Taint mencegah penjadwalan {{< glossary_tooltip text="Pod" term_id="pod" >}} pada {{< glossary_tooltip text="Node" term_id="node" >}} atau grup dari Node. + + + +Taint dan {{< glossary_tooltip text="toleransi" term_id="toleration" >}} bekerja sama untuk memastikan bahwa Pod tidak dijadwalkan ke Node yang tidak sesuai. Satu atau lebih taint dapat diterapkan pada Node. Sebuah Node seharusnya hanya menjadwalkan Pod dengan toleransi yang cocok untuk taint yang dikonfigurasi. diff --git a/content/id/docs/reference/glossary/toleration.md b/content/id/docs/reference/glossary/toleration.md new file mode 100644 index 0000000000..45a2ac7f7c --- /dev/null +++ b/content/id/docs/reference/glossary/toleration.md @@ -0,0 +1,17 @@ +--- +title: Toleransi (Toleration) +id: toleration +date: 2019-01-11 +full_link: /docs/concepts/scheduling-eviction/taint-and-toleration/ +short_description: > + Objek inti yang terdiri dari tiga properti yang diperlukan: _key_(kunci), _value_(nilai), dan _effect_(efek). Toleransi memungkinkan penjadwalan Pod pada Node atau grup dari Node yang memiliki taint yang cocok. +aka: +tags: +- core-object +- fundamental +--- + Objek inti yang terdiri dari tiga properti yang diperlukan: _key_(kunci), _value_(nilai), dan _effect_(efek). Toleransi memungkinkan penjadwalan Pod pada Node atau grup dari Node yang memiliki {{< glossary_tooltip text="taints" term_id="taint" >}} yang cocok. + + + +Toleransi dan {{< glossary_tooltip text="taints" term_id="taint" >}} bekerja sama untuk memastikan bahwa Pod tidak dijadwalkan ke Node yang tidak sesuai. Satu atau lebih taint dapat diterapkan pada Node. Sebuah Node seharusnya hanya menjadwalkan Pod dengan toleransi yang cocok untuk taint yang dikonfigurasi. diff --git a/content/id/docs/tasks/administer-cluster/sysctl-cluster.md b/content/id/docs/tasks/administer-cluster/sysctl-cluster.md index 9adbb50a9f..7120f087fa 100644 --- a/content/id/docs/tasks/administer-cluster/sysctl-cluster.md +++ b/content/id/docs/tasks/administer-cluster/sysctl-cluster.md @@ -156,7 +156,7 @@ Sangat disarankan untuk menggunakan Kubernetes [fitur _taints and toleration_](/ Pod dengan sysctl _unsafe_ akan gagal diluncurkan pada sembarang Node yang belum mengaktifkan kedua sysctl _unsafe_ secara eksplisit. Seperti halnya sysctl _node-level_ sangat disarankan untuk menggunakan [fitur _taints and toleration_](/docs/reference/generated/kubectl/kubectl-commands/#taint) atau -[pencemaran dalam Node](/docs/concepts/scheduling-eviction/taint-and-toleration/) +[pencemaran dalam Node](/id/docs/concepts/scheduling-eviction/taint-and-toleration/) untuk Pod dalam Node yang tepat. ## PodSecurityPolicy diff --git a/static/_redirects b/static/_redirects index db9f04df02..345c2707ca 100644 --- a/static/_redirects +++ b/static/_redirects @@ -99,6 +99,7 @@ /docs/concepts/cluster-administration/static-pod/ /docs/tasks/administer-cluster/static-pod/ 301 /docs/concepts/clusters/logging/ /docs/concepts/cluster-administration/logging/ 301 /docs/concepts/configuration/assign-pod-node/ /docs/concepts/scheduling-eviction/assign-pod-node/ 301 +/id/docs/concepts/configuration/assign-pod-node/ /id/docs/concepts/scheduling-eviction/assign-pod-node/ 301 /docs/concepts/configuration/container-command-arg/ /docs/tasks/inject-data-application/define-command-argument-container/ 301 /docs/concepts/configuration/container-command-args/ /docs/tasks/inject-data-application/define-command-argument-container/ 301 /docs/concepts/configuration/manage-compute-resources-container/ /docs/concepts/configuration/manage-resources-containers/ 301 @@ -106,6 +107,7 @@ /docs/concepts/configuration/scheduler-perf-tuning/ /docs/concepts/scheduling-eviction/scheduler-perf-tuning/ 301 /docs/concepts/configuration/scheduling-framework/ /docs/concepts/scheduling-eviction/scheduling-framework/ 301 /docs/concepts/configuration/taint-and-toleration/ /docs/concepts/scheduling-eviction/taint-and-toleration/ 301 +/id/docs/concepts/configuration/taint-and-toleration/ /id/docs/concepts/scheduling-eviction/taint-and-toleration/ 301 /docs/concepts/extend-kubernetes/extend-cluster/ /docs/concepts/extend-kubernetes/ 301 /docs/concepts/jobs/cron-jobs/ /docs/concepts/workloads/controllers/cron-jobs/ 301 /docs/concepts/jobs/run-to-completion-finite-workloads/ /docs/concepts/workloads/controllers/job/ 301 From c854d9e027708a087bf1b796033d70797c87e6c2 Mon Sep 17 00:00:00 2001 From: Zach Corleissen Date: Tue, 6 Apr 2021 11:32:36 -0700 Subject: [PATCH 050/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index b81c13a7f0..682489b227 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -1,7 +1,7 @@ --- layout: blog title: 'Annotating Kubernetes Services for Humans' -date: 2021-02-27 +date: 2021-04-06 slug: annotating-k8s-for-humans --- From 1ecc9948aa661c9f7140cfdacc75fbf410e36acc Mon Sep 17 00:00:00 2001 From: Zach Corleissen Date: Tue, 6 Apr 2021 11:32:45 -0700 Subject: [PATCH 051/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index 682489b227..2c28c00cb6 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -94,6 +94,6 @@ Historically, service catalogs have not been a key piece of infrastructure, as a ## Annotate your services now and thank yourself later Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Don't wait until something is on fire in production to start wishing you had implemented better metrics and also documented how to get in touch with the part of your organization that looks after it. -There's enormous benefits to building an effective “[version 0](https://www.getambassador.io/learn/kubernetes-glossary/version-0/)” of any service: a “[dancing skeleton](https://containerjournal.com/topics/container-management/dancing-skeleton-apis-and-microservices/)” application with a thin slice of complete functionality that can be deployed to production with a minimal yet effective continuous delivery pipeline. +There's enormous benefits to building an effective [version 0](https://www.getambassador.io/learn/kubernetes-glossary/version-0/) of any service: a “[dancing skeleton](https://containerjournal.com/topics/container-management/dancing-skeleton-apis-and-microservices/)” application with a thin slice of complete functionality that can be deployed to production with a minimal yet effective continuous delivery pipeline. Adding service annotations should be an essential part of your “version 0” for all of your services. Add them now, and you’ll thank yourself later. From 4079e588223d457c5c555cc1693412066a2f6d14 Mon Sep 17 00:00:00 2001 From: Zach Corleissen Date: Tue, 6 Apr 2021 11:43:04 -0700 Subject: [PATCH 052/218] Update content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md --- content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md index 2c28c00cb6..9e86770abe 100644 --- a/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md +++ b/content/en/blog/_posts/2021-02-27-annotating-k8s-for-humans.md @@ -1,7 +1,7 @@ --- layout: blog title: 'Annotating Kubernetes Services for Humans' -date: 2021-04-06 +date: 2021-04-12 slug: annotating-k8s-for-humans --- From 30c6e773e24d1a7f91d7b7c96f0c61add29f416d Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Mon, 15 Mar 2021 22:20:36 +0200 Subject: [PATCH 053/218] kubeadm: add guide for kubelet serving certs and metrics-server Include a new section of the kubeadm certificate management page to talk about kubelet serving certificates and how to make them signed. Also include a note about using secure connection with the metrics-server. --- .../tools/kubeadm/troubleshooting-kubeadm.md | 19 +++++- .../kubeadm/kubeadm-certs.md | 68 +++++++++++++++++++ 2 files changed, 86 insertions(+), 1 deletion(-) diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md index 58f4f4652a..4a8764a76b 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md @@ -408,7 +408,7 @@ be advised that this is modifying a design principle of the Linux distribution. ## `kubeadm upgrade plan` prints out `context deadline exceeded` error message -This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`. +This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`. This issue is fixed as of version 1.19. @@ -420,3 +420,20 @@ To workaround the issue, re-mount the `/var/lib/kubelet` directory after perform This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20. +## Cannot use the metrics-server securely in a kubeadm cluster + +In a kubeadm cluster, the [metrics-server](https://github.com/kubernetes-sigs/metrics-server) +can be used insecurely by passing the `--kubelet-insecure-tls` to it. This is not recommended for production clusters. + +If you want to use TLS between the metrics-server and the kubelet there is a problem, +since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors +on the side of the metrics-server: +``` +x509: certificate signed by unknown authority +x509: certificate is valid for IP-foo not IP-bar +``` + +See [Enabling signed kubelet serving certificates](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs) +to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates. + +Also see [How to run the metrics-server securely](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-to-run-metrics-server-securely). diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index dc7af4a329..62e66d1a8f 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -207,3 +207,71 @@ After a certificate is signed using your preferred method, the certificate and t Kubeadm does not support rotation or replacement of CA certificates out of the box. For more information about manual rotation or replacement of CA, see [manual rotation of CA certificates](/docs/tasks/tls/manual-rotation-of-ca-certificates/). + +## Enabling signed kubelet serving certificates {#kubelet-serving-certs} + +By default the kubelet serving certificate deployed by kubeadm is self-signed. +This means a connection from external services like the +[metrics-server](https://github.com/kubernetes-sigs/metrics-server) to a +kubelet cannot be secured with TLS. + +To configure the kubelets in a new kubeadm cluster to obtain properly signed serving +certificates you must pass the following minimal configuration to `kubeadm init`: + +```yaml +apiVersion: kubeadm.k8s.io/v1beta2 +kind: ClusterConfiguration +--- +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +serverTLSBootstrap: true +``` + +If you have already created the cluster you must adapt it by doing the following: + - Find and edit the `kubelet-config-{{< skew latestVersion >}}` ConfigMap in the `kube-system` namespace. +In that ConfigMap, the `config` key has a +[KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) +document as its value. Edit the KubeletConfiguration document to set `serverTLSBootstrap: true`. +- On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml` +and restart the kubelet with `systemctl restart kubelet` + +The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet serving +certificates by requesting them from the `certificates.k8s.io` API. One known limitation +is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically +approved by the default signer in the kube-controller-manager - +[`kubernetes.io/kubelet-serving`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers). +This will require action from the user or a third party controller. + +These CSRs can be viewed using: + +```shell +kubectl get csr +NAME AGE SIGNERNAME REQUESTOR CONDITION +csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending +csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending +``` + +To approve them you can do the following: +```shell +kubectl certificate approve +``` + +By default, these serving certificate will expire after one year. Kubeadm sets the +`KubeletConfiguration` field `rotateCertificates` to `true`, which means that close +to expiration a new set of CSRs for the serving certificates will be created and must +be approved to complete the rotation. To understand more see +[Certificate Rotation](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation). + +If you are looking for a solution for automatic approval of these CSRs it is recommended +that you contact your cloud provider and ask if they have a CSR signer that verifies +the node identity with an out of band mechanism. + +{{% thirdparty-content %}} + +Third party custom controllers can be used: +- [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp) + +Such a controller is not a secure mechanism unless it not only verifies the CommonName +in the CSR but also verifies the requested IPs and domain names. This would prevent +a malicious actor that has access to a kubelet client certificate to create +CSRs requesting serving certificates for any IP or domain name. From 108149fa2f8fa9c6c244ce6a551df926d8062df1 Mon Sep 17 00:00:00 2001 From: Qiming Teng Date: Thu, 18 Feb 2021 15:41:00 +0800 Subject: [PATCH 054/218] Add WebhookAdmission reference This is a reference for WebhookAdmission config generated from kubernetes-sigs/reference-docs/genref tool. More specifically, it is generated using the following command: ```shell ./genref -include apiserver-webhookadmission ``` --- .../docs/concepts/configuration/configmap.md | 2 +- content/en/docs/reference/_index.md | 9 +--- .../extensible-admission-controllers.md | 4 +- .../apiserver-webhookadmission.v1.md | 46 +++++++++++++++++++ .../tasks/debug-application-cluster/audit.md | 4 +- 5 files changed, 54 insertions(+), 11 deletions(-) create mode 100644 content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md diff --git a/content/en/docs/concepts/configuration/configmap.md b/content/en/docs/concepts/configuration/configmap.md index b5f5bd8d28..c0d21eaf16 100644 --- a/content/en/docs/concepts/configuration/configmap.md +++ b/content/en/docs/concepts/configuration/configmap.md @@ -224,7 +224,7 @@ When a ConfigMap currently consumed in a volume is updated, projected keys are e The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, the kubelet uses its local cache for getting the current value of the ConfigMap. The type of the cache is configurable using the `ConfigMapAndSecretChangeDetectionStrategy` field in -the [KubeletConfiguration struct](/docs/reference/config-api/kubelet-config.v1beta1/)). +the [KubeletConfiguration struct](/docs/reference/config-api/kubelet-config.v1beta1/). A ConfigMap can be either propagated by watch (default), ttl-based, or by redirecting all requests directly to the API server. As a result, the total delay from the moment when the ConfigMap is updated to the moment diff --git a/content/en/docs/reference/_index.md b/content/en/docs/reference/_index.md index d8ed48db8b..55e9a91d9f 100644 --- a/content/en/docs/reference/_index.md +++ b/content/en/docs/reference/_index.md @@ -73,16 +73,11 @@ operator to use or manage a cluster. * [kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) * [kube-scheduler configuration (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) +* [kube-scheduler policy reference (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) * [kube-proxy configuration (v1alpha1)](/docs/reference/config-api/kube-proxy-config.v1alpha1/) * [`audit.k8s.io/v1` API](/docs/reference/config-api/apiserver-audit.v1/) - -## Config APIs - * [Client authentication API (v1beta1)](/docs/reference/config-api/client-authentication.v1beta1/) - -## Config APIs - -* [kube-scheduler policy reference (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) +* [WebhookAdmission configuration (v1)](/docs/reference/config-api/apiserver-webhookadmission.v1/) ## Design Docs diff --git a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md index 018195f817..051b81d21b 100644 --- a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -147,7 +147,7 @@ webhooks: {{< /tabs >}} The scope field specifies if only cluster-scoped resources ("Cluster") or namespace-scoped -resources ("Namespaced") will match this rule. "*" means that there are no scope restrictions. +resources ("Namespaced") will match this rule. "∗" means that there are no scope restrictions. {{< note >}} When using `clientConfig.service`, the server cert must be valid for @@ -225,7 +225,7 @@ plugins: {{< /tabs >}} For more information about `AdmissionConfiguration`, see the -[AdmissionConfiguration schema](https://github.com/kubernetes/kubernetes/blob/v1.17.0/staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1/types.go#L27). +[AdmissionConfiguration (v1) reference](/docs/reference/config-api/apiserver-webhookadmission.v1/). See the [webhook configuration](#webhook-configuration) section for details about each config field. * In the kubeConfig file, provide the credentials: diff --git a/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md b/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md new file mode 100644 index 0000000000..fb45ca7b1a --- /dev/null +++ b/content/en/docs/reference/config-api/apiserver-webhookadmission.v1.md @@ -0,0 +1,46 @@ +--- +title: WebhookAdmission Configuration (v1) +content_type: tool-reference +package: apiserver.config.k8s.io/v1 +auto_generated: true +--- +Package v1 is the v1 version of the API. + +## Resource Types + + +- [WebhookAdmission](#apiserver-config-k8s-io-v1-WebhookAdmission) + + + + +## `WebhookAdmission` {#apiserver-config-k8s-io-v1-WebhookAdmission} + + + + + +WebhookAdmission provides configuration for the webhook admission controller. + + + + + + + + + + + + + + + + + +
FieldDescription
apiVersion
string		apiserver.config.k8s.io/v1
kind
string		WebhookAdmission
kubeConfigFile [Required]
+string +		 + KubeConfigFile is the path to the kubeconfig file.
+ + diff --git a/content/en/docs/tasks/debug-application-cluster/audit.md b/content/en/docs/tasks/debug-application-cluster/audit.md index 2f8e1c9875..c44caf66b5 100644 --- a/content/en/docs/tasks/debug-application-cluster/audit.md +++ b/content/en/docs/tasks/debug-application-cluster/audit.md @@ -251,5 +251,7 @@ By default truncate is disabled in both `webhook` and `log`, a cluster administr ## {{% heading "whatsnext" %}} * Learn about [Mutating webhook auditing annotations](/docs/reference/access-authn-authz/extensible-admission-controllers/#mutating-webhook-auditing-annotations). -* Read the [reference for `audit.k8s.io` API group](/docs/reference/config-api/apiserver-audit.v1/). +* Learn more about [`Event`](/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event) + and the [`Policy`](/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Policy) + resource types by reading the Audit configuration reference. From 47df9fdab0c596eefc62764f0725c05db76f1937 Mon Sep 17 00:00:00 2001 From: Alay Patel Date: Wed, 24 Mar 2021 12:34:34 -0400 Subject: [PATCH 055/218] Blog: CronJob reaches GA --- .../controller-flowchart.svg | 1 + .../2021-04-08-cronjob-reaches-ga/index.md | 105 ++++++++++++++++++ .../performance-impact-graph.svg | 1 + 3 files changed, 107 insertions(+) create mode 100644 content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/controller-flowchart.svg create mode 100644 content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/index.md create mode 100644 content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/performance-impact-graph.svg diff --git a/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/controller-flowchart.svg b/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/controller-flowchart.svg new file mode 100644 index 0000000000..9357c89d40 --- /dev/null +++ b/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/controller-flowchart.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/index.md b/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/index.md new file mode 100644 index 0000000000..d6a245cbf7 --- /dev/null +++ b/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/index.md @@ -0,0 +1,105 @@ +--- +layout: blog +title: 'Kubernetes 1.21: CronJob Reaches GA' +date: 2021-04-09 +slug: kubernetes-release-1.21-cronjob-ga +--- + + **Authors:** Alay Patel (Red Hat), and Maciej Szulik (Red Hat) + +In Kubernetes v1.21, the +[CronJob](/docs/concepts/workloads/controllers/cron-jobs/) resource +reached general availability (GA). We've also substantially improved the +performance of CronJobs since Kubernetes v1.19, by implementing a new +controller. + +In Kubernetes v1.20 we launched a revised v2 controller for CronJobs, +initially as an alpha feature. Kubernetes 1.21 uses the newer controller by +default, and the CronJob resource itself is now GA (group version: `batch/v1`). + +In this article, we'll take you through the driving forces behind this new +development, give you a brief description of controller design for core +Kubernetes, and we'll outline what you will gain from this improved controller. + +The driving force behind promoting the API was Kubernetes' policy choice to +[ensure APIs move beyond beta](/blog/2020/08/21/moving-forward-from-beta/). +That policy aims to prevent APIs from being stuck in a “permanent beta” state. +Over the years the old CronJob controller implementation had received healthy +feedback from the community, with reports of several widely recognized +[issues](https://github.com/kubernetes/kubernetes/issues/82659). + +If the beta API for CronJob was to be supported as GA, the existing controller +code would need substantial rework. Instead, the SIG Apps community decided +to introduce a new controller and gradually replace the old one. + +## How do controllers work? + +Kubernetes [controllers](/docs/concepts/architecture/controller/) are control +loops that watch the state of resource(s) in your cluster, then make or +request changes where needed. Each controller tries to move part of the +current cluster state closer to the desired state. + +The v1 CronJob controller works by performing a periodic poll and sweep of all +the CronJob objects in your cluster, in order to act on them. It is a single +worker implementation that gets all CronJobs every 10 seconds, iterates over +each one of them, and syncs them to their desired state. This was the default +way of doing things almost 5 years ago when the controller was initially +written. In hindsight, we can certainly say that such an approach can +overload the API server at scale. + +These days, every core controller in kubernetes must follow the guidelines +described in [Writing Controllers](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md#readme). +Among many details, that document prescribes using +[shared informers](https://www.cncf.io/blog/2019/10/15/extend-kubernetes-via-a-shared-informer/) +to “receive notifications of adds, updates, and deletes for a particular +resource”. Upon any such events, the related object(s) is placed in a queue. +Workers pull items from the queue and process them one at a time. This +approach ensures consistency and scalability. + +The picture below shows the flow of information from kubernetes API server, +through shared informers and queue, to the main part of a controller - a +reconciliation loop which is responsible for performing the core functionality. + +![Controller flowchart](controller-flowchart.svg) + +The CronJob controller V2 uses a queue that implements the DelayingInterface to +handle the scheduling aspect. This queue allows processing an element after a +specific time interval. Every time there is a change in a CronJob or its related +Jobs, the key that represents the CronJob is pushed to the queue. The main +handler pops the key, processes the CronJob, and after completion +pushes the key back into the queue for the next scheduled time interval. This is +immediately a more performant implementation, as it no longer requires a linear +scan of all the CronJobs. On top of that, this controller can be scaled by +increasing the number of workers processing the CronJobs in parallel. + +## Performance impact of the new controller {#performance-impact} + +In order to test the performance difference of the two controllers a VM instance +with 128 GiB RAM and 64 vCPUs was used to set up a single node Kubernetes cluster. +Initially, a sample workload was created with 20 CronJob instances with a schedule +to run every minute, and 2100 CronJobs running every 20 hours. Additionally, +over the next few minutes we added 1000 CronJobs with a schedule to run every +20 hours, until we reached a total of 5120 CronJobs. + +![Visualization of performance](performance-impact-graph.svg) + +We observed that for every 1000 CronJobs added, the old controller used +around 90 to 120 seconds more wall-clock time to schedule 20 Jobs every cycle. +That is, at 5120 CronJobs, the old controller took approximately 9 minutes +to create 20 Jobs. Hence, during each cycle, about 8 schedules were missed. +The new controller, implemented with architectural change explained above, +created 20 Jobs without any delay, even when we created an additional batch +of 1000 CronJobs reaching a total of 6120. + +As a closing remark, the new controller exposes a histogram metric +`cronjob_controller_cronjob_job_creation_skew_duration_seconds` which helps +monitor the time difference between when a CronJob is meant to run and when +the actual Job is created. + +Hopefully the above description is a sufficient argument to follow the +guidelines and standards set in the Kubernetes project, even for your own +controllers. As mentioned before, the new controller is on by default starting +from Kubernetes v1.21; if you want to check it out in the previous release (1.20), +you can enable the `CronJobControllerV2` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +for the kube-controller-manger: `--feature-gate="CronJobControllerV2=true"`. diff --git a/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/performance-impact-graph.svg b/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/performance-impact-graph.svg new file mode 100644 index 0000000000..976b428d91 --- /dev/null +++ b/content/en/blog/_posts/2021-04-08-cronjob-reaches-ga/performance-impact-graph.svg @@ -0,0 +1 @@ + \ No newline at end of file From 0c988b2dcb2c891e49702fe04c178cc61e3ff59d Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Wed, 7 Apr 2021 15:49:24 -0700 Subject: [PATCH 056/218] Correct dockershim runtime handler support --- content/en/docs/concepts/containers/runtime-class.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/containers/runtime-class.md b/content/en/docs/concepts/containers/runtime-class.md index abfec1ef6c..6af609636e 100644 --- a/content/en/docs/concepts/containers/runtime-class.md +++ b/content/en/docs/concepts/containers/runtime-class.md @@ -109,7 +109,8 @@ For more details on setting up CRI runtimes, see [CRI installation](/docs/setup/ #### dockershim -Kubernetes built-in dockershim CRI does not support runtime handlers. +RuntimeClasses with dockershim must set the runtime handler to `docker`. Dockershim does not support +custom configurable runtime handlers. #### {{< glossary_tooltip term_id="containerd" >}} @@ -163,7 +164,7 @@ Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/). {{< feature-state for_k8s_version="v1.18" state="beta" >}} You can specify _overhead_ resources that are associated with running a Pod. Declaring overhead allows -the cluster (including the scheduler) to account for it when making decisions about Pods and resources. +the cluster (including the scheduler) to account for it when making decisions about Pods and resources. To use Pod overhead, you must have the PodOverhead [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled (it is on by default). From aa7d13a5bf29dcbc638ac5054a340614e0a3d8c7 Mon Sep 17 00:00:00 2001 From: caodonghui Date: Thu, 8 Apr 2021 18:29:02 +0800 Subject: [PATCH 057/218] [zh]update configure-upgrade-etcd.md --- .../configure-upgrade-etcd.md | 567 ++++++++++++------ 1 file changed, 367 insertions(+), 200 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/configure-upgrade-etcd.md b/content/zh/docs/tasks/administer-cluster/configure-upgrade-etcd.md index 749aabc73d..c64815d38b 100644 --- a/content/zh/docs/tasks/administer-cluster/configure-upgrade-etcd.md +++ b/content/zh/docs/tasks/administer-cluster/configure-upgrade-etcd.md @@ -18,12 +18,8 @@ content_type: task {{< glossary_definition term_id="etcd" length="all" >}} - - - ## {{% heading "prerequisites" %}} - {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} @@ -35,13 +31,21 @@ content_type: task * Run etcd as a cluster of odd members. -* etcd is a leader-based distributed system. Ensure that the leader periodically send heartbeats on time to all followers to keep the cluster stable. +* etcd is a leader-based distributed system. Ensure that the leader + periodically send heartbeats on time to all followers to keep the cluster + stable. * Ensure that no resource starvation occurs. - Performance and stability of the cluster is sensitive to network and disk IO. Any resource starvation can lead to heartbeat timeout, causing instability of the cluster. An unstable etcd indicates that no leader is elected. Under such circumstances, a cluster cannot make any changes to its current state, which implies no new pods can be scheduled. + Performance and stability of the cluster is sensitive to network and disk + I/O. Any resource starvation can lead to heartbeat timeout, causing instability + of the cluster. An unstable etcd indicates that no leader is elected. Under + such circumstances, a cluster cannot make any changes to its current state, + which implies no new pods can be scheduled. -* Keeping stable etcd clusters is critical to the stability of Kubernetes clusters. Therefore, run etcd clusters on dedicated machines or isolated environments for [guaranteed resource requirements](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations). +* Keeping etcd clusters stable is critical to the stability of Kubernetes + clusters. Therefore, run etcd clusters on dedicated machines or isolated + environments for [guaranteed resource requirements](https://etcd.io/docs/current/op-guide/hardware/). * The minimum recommended version of etcd to run in production is `3.2.10+`. --> @@ -53,16 +57,23 @@ content_type: task * 确保不发生资源不足。 - 集群的性能和稳定性对网络和磁盘 IO 非常敏感。任何资源匮乏都会导致心跳超时，从而导致集群的不稳定。不稳定的情况表明没有选出任何主节点。在这种情况下，集群不能对其当前状态进行任何更改，这意味着不能调度新的 pod。 + 集群的性能和稳定性对网络和磁盘 I/O 非常敏感。任何资源匮乏都会导致心跳超时， + 从而导致集群的不稳定。不稳定的情况表明没有选出任何主节点。 + 在这种情况下，集群不能对其当前状态进行任何更改，这意味着不能调度新的 pod。 -* 保持稳定的 etcd 集群对 Kubernetes 集群的稳定性至关重要。因此，请在专用机器或隔离环境上运行 etcd 集群，以满足[所需资源需求](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations)。 +* 保持 etcd 集群的稳定对 Kubernetes 集群的稳定性至关重要。 + 因此，请在专用机器或隔离环境上运行 etcd 集群，以满足 + [所需资源需求](https://etcd.io/docs/current/op-guide/hardware/)。 * 在生产中运行的 etcd 的最低推荐版本是 `3.2.10+`。 ## 资源要求 -使用有限的资源运行 etcd 只适合测试目的。为了在生产中部署，需要先进的硬件配置。在生产中部署 etcd 之前，请查看[所需资源参考文档](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#example-hardware-configurations)。 +使用有限的资源运行 etcd 只适合测试目的。为了在生产中部署，需要先进的硬件配置。 +在生产中部署 etcd 之前，请查看 +[所需资源参考文档](https://etcd.io/docs/current/op-guide/hardware/#example-hardware-configurations)。 ## 启动 etcd 集群 @@ -83,13 +96,15 @@ Use a single-node etcd cluster only for testing purpose. 1. Run the following: - ```sh - ./etcd --listen-client-urls=http://$PRIVATE_IP:2379 --advertise-client-urls=http://$PRIVATE_IP:2379 - ``` + ```sh + etcd --listen-client-urls=http://$PRIVATE_IP:2379 \ + --advertise-client-urls=http://$PRIVATE_IP:2379 + ``` -2. Start Kubernetes API server with the flag `--etcd-servers=$PRIVATE_IP:2379`. +2. Start the Kubernetes API server with the flag + `--etcd-servers=$PRIVATE_IP:2379`. - Replace `PRIVATE_IP` with your etcd client IP. + Make sure `PRIVATE_IP` is set to your etcd client IP. --> ### 单节点 etcd 集群 @@ -97,53 +112,70 @@ Use a single-node etcd cluster only for testing purpose. 1. 运行以下命令： - ```sh - ./etcd --listen-client-urls=http://$PRIVATE_IP:2379 --advertise-client-urls=http://$PRIVATE_IP:2379 - ``` + ```sh + etcd --listen-client-urls=http://$PRIVATE_IP:2379 \ + --advertise-client-urls=http://$PRIVATE_IP:2379 + ``` 2. 使用参数 `--etcd-servers=$PRIVATE_IP:2379` 启动 Kubernetes API 服务器。 - 使用您 etcd 客户端 IP 替换 `PRIVATE_IP`。 + 确保将 `PRIVATE_IP` 设置为etcd客户端 IP。 ### 多节点 etcd 集群 -为了耐用性和高可用性，在生产中将以多节点集群的方式运行 etcd，并且定期备份。建议在生产中使用五个成员的集群。有关该内容的更多信息，请参阅[常见问题文档](https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-is-failure-tolerance)。 +为了耐用性和高可用性，在生产中将以多节点集群的方式运行 etcd，并且定期备份。 +建议在生产中使用五个成员的集群。 +有关该内容的更多信息，请参阅 +[常见问题文档](https://etcd.io/docs/current/faq/#what-is-failure-tolerance)。 -可以通过静态成员信息或动态发现的方式配置 etcd 集群。有关集群的详细信息，请参阅 [etcd 集群文档](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md)。 +可以通过静态成员信息或动态发现的方式配置 etcd 集群。 +有关集群的详细信息，请参阅 +[etcd 集群文档](https://etcd.io/docs/current/op-guide/clustering/)。 -例如，考虑运行以下客户端 URL 的五个成员的 etcd 集群：`http://$IP1:2379`，`http://$IP2:2379`，`http://$IP3:2379`，`http://$IP4:2379` 和 `http://$IP5:2379`。要启动 Kubernetes API 服务器： +例如，考虑运行以下客户端 URL 的五个成员的 etcd 集群：`http://$IP1:2379`， +`http://$IP2:2379`，`http://$IP3:2379`，`http://$IP4:2379` 和 `http://$IP5:2379`。 +要启动 Kubernetes API 服务器： 1. 运行以下命令： - ```sh - ./etcd --listen-client-urls=http://$IP1:2379, http://$IP2:2379, http://$IP3:2379, http://$IP4:2379, http://$IP5:2379 --advertise-client-urls=http://$IP1:2379, http://$IP2:2379, http://$IP3:2379, http://$IP4:2379, http://$IP5:2379 - ``` + ```shell + etcd --listen-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 --advertise-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 + ``` -2. 使用参数 `--etcd-servers=$IP1:2379, $IP2:2379, $IP3:2379, $IP4:2379, $IP5:2379` 启动 Kubernetes API 服务器。 +2. 使用参数 `--etcd-servers=$IP1:2379,$IP2:2379,$IP3:2379,$IP4:2379,$IP5:2379` + 启动 Kubernetes API 服务器。 - 使用您 etcd 客户端 IP 地址替换 `IP`。 + 确保将 `IP` 变量设置为客户端 IP 地址。 ## 安全的 etcd 集群 -对 etcd 的访问相当于集群中的 root 权限，因此理想情况下只有 API 服务器才能访问它。考虑到数据的敏感性，建议只向需要访问 etcd 集群的节点授予权限。 +对 etcd 的访问相当于集群中的 root 权限，因此理想情况下只有 API 服务器才能访问它。 +考虑到数据的敏感性，建议只向需要访问 etcd 集群的节点授予权限。 -想要确保 etcd 的安全，可以设置防火墙规则或使用 etcd 提供的安全特性，这些安全特性依赖于 x509 公钥基础设施（PKI）。首先，通过生成密钥和证书对来建立安全的通信通道。 -例如，使用密钥对 `peer.key` 和 `peer.cert` 来保护 etcd 成员之间的通信，而 `client.cert` 和 `client.cert` 用于保护 etcd 与其客户端之间的通信。请参阅 etcd 项目提供的[示例脚本](https://github.com/coreos/etcd/tree/master/hack/tls-setup)，以生成用于客户端身份验证的密钥对和 CA 文件。 +想要确保 etcd 的安全，可以设置防火墙规则或使用 etcd 提供的安全特性，这些安全特性依赖于 x509 公钥基础设施（PKI）。 +首先，通过生成密钥和证书对来建立安全的通信通道。 +例如，使用密钥对 `peer.key` 和 `peer.cert` 来保护 etcd 成员之间的通信， +而 `client.key` 和 `client.cert` 用于保护 etcd 与其客户端之间的通信。 +请参阅 etcd 项目提供的[示例脚本](https://github.com/coreos/etcd/tree/master/hack/tls-setup)， +以生成用于客户端身份验证的密钥对和 CA 文件。 ### 安全通信 -若要使用安全对等通信对 etcd 进行配置，请指定参数 `--peer-key-file=peer.key` 和 `--peer-cert-file=peer.cert`，并使用 https 作为 URL 模式。 +若要使用安全对等通信对 etcd 进行配置，请指定参数 `--peer-key-file=peer.key` +和 `--peer-cert-file=peer.cert`，并使用 HTTPS 作为 URL 模式。 -类似地，要使用安全客户端通信对 etcd 进行配置，请指定参数 `--key-file=k8sclient.key` 和 `--cert-file=k8sclient.cert`，并使用 https 作为 URL 模式。 +类似地，要使用安全客户端通信对 etcd 进行配置，请指定参数 `--key-file=k8sclient.key` +和 `--cert-file=k8sclient.cert`，并使用 HTTPS 作为 URL 模式。 +使用安全通信的客户端命令的示例： + +``` +ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + member list +``` ### 限制 etcd 集群的访问 配置安全通信后，将 etcd 集群的访问限制在 Kubernetes API 服务器上。使用 TLS 身份验证来完成此任务。 -例如，考虑由 CA `etcd.ca` 信任的密钥对 `k8sclient.key` 和 `k8sclient.cert`。当 etcd 配置为 `--client-cert-auth` 和 TLS 时，它使用系统 CA 或由 `--trusted-ca-file` 参数传入的 CA 验证来自客户端的证书。 +例如，考虑由 CA `etcd.ca` 信任的密钥对 `k8sclient.key` 和 `k8sclient.cert`。 +当 etcd 配置为 `--client-cert-auth` 和 TLS 时，它使用系统 CA 或由 `--trusted-ca-file` 参数传入的 CA 验证来自客户端的证书。 指定参数 `--client-cert-auth=true` 和 `--trusted-ca-file=etcd.ca` 将限制对具有证书 `k8sclient.cert` 的客户端的访问。 -一旦正确配置了 etcd，只有具有有效证书的客户端才能访问它。要让 Kubernetes API 服务器访问，可以使用参数 `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` 和 `--etcd-cafile=ca.cert` 配置它。 +一旦正确配置了 etcd，只有具有有效证书的客户端才能访问它。要让 Kubernetes API 服务器访问， +可以使用参数 `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` 和 `--etcd-cafile=ca.cert` 配置。 {{< note >}} -Kubernetes 目前不支持 etcd 身份验证。想要了解更多信息，请参阅相关的问题[支持 etcd v2 的基本认证](https://github.com/kubernetes/kubernetes/issues/23398)。 +Kubernetes 目前不支持 etcd 身份验证。 +想要了解更多信息，请参阅相关的问题 +[支持 etcd v2 的基本认证](https://github.com/kubernetes/kubernetes/issues/23398)。 {{< /note >}} ## 替换失败的 etcd 成员 -etcd 集群通过容忍少数成员故障实现高可用性。但是，要改善集群的整体健康状况，请立即替换失败的成员。当多个成员失败时，逐个替换它们。替换失败成员需要两个步骤：删除失败成员和添加新成员。 +etcd 集群通过容忍少数成员故障实现高可用性。 +但是，要改善集群的整体健康状况，请立即替换失败的成员。当多个成员失败时，逐个替换它们。 +替换失败成员需要两个步骤：删除失败成员和添加新成员。 -虽然 etcd 在内部保留唯一的成员 ID，但建议为每个成员使用唯一的名称，以避免人为错误。例如，考虑一个三成员的 etcd 集群。让 URL 为：member1=http://10.0.0.1， member2=http://10.0.0.2 和 member3=http://10.0.0.3。当 member1 失败时，将其替换为 member4=http://10.0.0.4。 +虽然 etcd 在内部保留唯一的成员 ID，但建议为每个成员使用唯一的名称，以避免人为错误。 +例如，考虑一个三成员的 etcd 集群。让 URL 为：`member1=http://10.0.0.1`， `member2=http://10.0.0.2` +和 `member3=http://10.0.0.3`。当 `member1` 失败时，将其替换为 `member4=http://10.0.0.4`。 -1. 获取失败的 member1 的成员 ID： +1. 获取失败的 `member1` 的成员 ID： - `etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list` + ```shell + etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list + ``` - 显示以下信息： + 显示以下信息： - 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 - 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 - fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 + ```console + 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 + 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 + fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 + ``` 2. 移除失败的成员 - `etcdctl member remove 8211f1d0f64f3269` + ```shell + etcdctl member remove 8211f1d0f64f3269 + ``` - 显示以下信息： + 显示以下信息： - Removed member 8211f1d0f64f3269 from cluster + ```console + Removed member 8211f1d0f64f3269 from cluster + ``` 3. 增加新成员： - `./etcdctl member add member4 --peer-urls=http://10.0.0.4:2380` + ```shell + etcdctl member add member4 --peer-urls=http://10.0.0.4:2380 + ``` - 显示以下信息： + 显示以下信息： - Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 + ```console + Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 + ``` 4. 在 IP 为 `10.0.0.4` 的机器上启动新增加的成员： - export ETCD_NAME="member4" - export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" - export ETCD_INITIAL_CLUSTER_STATE=existing - etcd [flags] + ```shell + export ETCD_NAME="member4" + export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" + export ETCD_INITIAL_CLUSTER_STATE=existing + etcd [flags] + ``` 5. 做以下事情之一： - 1. 更新其 `--etcd-servers` 参数，使 Kubernetes 知道配置进行了更改，然后重新启动 Kubernetes API 服务器。 + 1. 更新 Kubernetes API 服务器的 `--etcd-servers` 参数，使 Kubernetes 知道配置进行了更改，然后重新启动 Kubernetes API 服务器。 2. 如果在 deployment 中使用了负载均衡，更新负载均衡配置。 -有关集群重新配置的详细信息，请参阅 [etcd 重构文档](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member)。 +有关集群重新配置的详细信息，请参阅 [etcd 重构文档](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member)。 ## 备份 etcd 集群 -所有 Kubernetes 对象都存储在 etcd 上。定期备份 etcd 集群数据对于在灾难场景（例如丢失所有主节点）下恢复 Kubernetes 集群非常重要。快照文件包含所有 Kubernetes 状态和关键信息。为了保证敏感的 Kubernetes 数据的安全，可以对快照文件进行加密。 +所有 Kubernetes 对象都存储在 etcd 上。定期备份 etcd 集群数据对于在灾难场景（例如丢失所有控制平面节点）下恢复 Kubernetes 集群非常重要。 +快照文件包含所有 Kubernetes 状态和关键信息。为了保证敏感的 Kubernetes 数据的安全，可以对快照文件进行加密。 备份 etcd 集群可以通过两种方式完成：etcd 内置快照和卷快照。 ### 内置快照 -etcd 支持内置快照，因此备份 etcd 集群很容易。快照可以从使用 `etcdctl snapshot save` 命令的活动成员中获取，也可以通过从 etcd [数据目录](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir)复制 `member/snap/db` 文件，该 etcd 数据目录目前没有被 etcd 进程使用。获取快照通常不会影响成员的性能。 +etcd 支持内置快照。快照可以从使用 `etcdctl snapshot save` 命令的活动成员中获取， +也可以通过从 etcd [数据目录](https://etcd.io/docs/current/op-guide/configuration/#--data-dir) +复制 `member/snap/db` 文件，该 etcd 数据目录目前没有被 etcd 进程使用。获取快照不会影响成员的性能。 下面是一个示例，用于获取 `$ENDPOINT` 所提供的键空间的快照到文件 `snapshotdb`： -```sh +```shell ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshotdb -# exit 0 +``` -# verify the snapshot +验证快照: + +```shell ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshotdb +``` + +```console +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ @@ -361,54 +516,137 @@ ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshotdb ### 卷快照 如果 etcd 运行在支持备份的存储卷（如 Amazon Elastic Block 存储）上，则可以通过获取存储卷的快照来备份 etcd 数据。 +### 使用 etcdctl 选项的快照 + +我们还可以使用 etcdctl 提供的各种选项来拍摄快照。例如： + +```shell +ETCDCTL_API=3 etcdctl -h +``` + +列出 etcdctl 可用的各种选项。例如，你可以通过指定端点，证书等来拍摄快照，如下所示： + +```shell +ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \ + --cacert= --cert= --key= \ + snapshot save +``` +可以从 etcd Pod 的描述中获得 `trusted-ca-file`, `cert-file` 和 `key-file` 。 + ## 扩展 etcd 集群 -通过交换性能，扩展 etcd 集群可以提高可用性。缩放不会提高集群性能和能力。一般情况下不要扩大或缩小 etcd 集群的集合。不要为 etcd 集群配置任何自动缩放组。强烈建议始终在任何官方支持的规模上运行生产 Kubernetes 集群时使用静态的五成员 etcd 集群。 +通过交换性能，扩展 etcd 集群可以提高可用性。缩放不会提高集群性能和能力。 +一般情况下不要扩大或缩小 etcd 集群的集合。不要为 etcd 集群配置任何自动缩放组。 +强烈建议始终在任何官方支持的规模上运行生产 Kubernetes 集群时使用静态的五成员 etcd 集群。 -合理的扩展是在需要更高可靠性的情况下，将三成员集群升级为五成员集群。请参阅 [etcd 重新配置文档](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member)以了解如何将成员添加到现有集群中的信息。 +合理的扩展是在需要更高可靠性的情况下，将三成员集群升级为五成员集群。 +请参阅 [etcd 重新配置文档](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member) +以了解如何将成员添加到现有集群中的信息。 ## 恢复 etcd 集群 -etcd 支持从 [major.minor](http://semver.org/) 或其他不同 patch 版本的 etcd 进程中获取的快照进行恢复。还原操作用于恢复失败的集群的数据。 +etcd 支持从 [major.minor](http://semver.org/) 或其他不同 patch 版本的 etcd 进程中获取的快照进行恢复。 +还原操作用于恢复失败的集群的数据。 -在启动还原操作之前，必须有一个快照文件。它可以是来自以前备份操作的快照文件，也可以是来自剩余[数据目录](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir)的快照文件。 -有关从快照文件还原集群的详细信息和示例，请参阅 [etcd 灾难恢复文档](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster)。 +在启动还原操作之前，必须有一个快照文件。它可以是来自以前备份操作的快照文件， +也可以是来自剩余[数据目录]( https://etcd.io/docs/current/op-guide/configuration/#--data-dir)的快照文件。 +例如： + +```shell +ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 snapshot restore snapshotdb +``` + +有关从快照文件还原集群的详细信息和示例，请参阅 +[etcd 灾难恢复文档](https://etcd.io/docs/current/op-guide/recovery/#restoring-a-cluster)。 如果还原的集群的访问 URL 与前一个集群不同，则必须相应地重新配置 Kubernetes API 服务器。 在本例中，使用参数 `--etcd-servers=$NEW_ETCD_CLUSTER` 而不是参数 `--etcd-servers=$OLD_ETCD_CLUSTER` 重新启动 Kubernetes API 服务器。 @@ -420,82 +658,11 @@ etcd 支持从 [major.minor](http://semver.org/) 或其他不同 patch 版本的 {{< note >}} 如果集群中正在运行任何 API 服务器，则不应尝试还原 etcd 的实例。相反，请按照以下步骤还原 etcd： -- 停止 *所有* kube-apiserver 实例 +- 停止 *所有* API 服务实例 - 在所有 etcd 实例中恢复状态 -- 重启所有 kube-apiserver 实例 +- 重启所有 API 服务实例 -我们还建议重启所有组件（例如 kube-scheduler、kube-controller-manager、kubelet），以确保它们不会 +我们还建议重启所有组件（例如 `kube-scheduler`、`kube-controller-manager`、`kubelet`），以确保它们不会 依赖一些过时的数据。请注意，实际中还原会花费一些时间。 在还原过程中，关键组件将丢失领导锁并自行重启。 -{{< note >}} - - -## 升级和回滚 etcd 集群 - -从 Kubernetes v1.13.0 开始，不在支持 etcd2 作为新的或现有 Kubernetes 集群的后端。Kubernetes 支持 etcd2 和 etcd3 的时间表如下： - -- Kubernetes v1.0: 仅限 etcd2 -- Kubernetes v1.5.1: 添加了 etcd3 支持，新的集群仍默认为 etcd2 -- Kubernetes v1.6.0: 使用 `kube-up.sh` 创建的新集群默认为 etcd3，而 `kube-apiserver` 默认为 etcd3 -- Kubernetes v1.9.0: 宣布弃用 etcd2 存储后端 -- Kubernetes v1.13.0: 删除了 etcd2 存储后端，`kube-apiserver` 将拒绝以 `--storage-backend = etcd2` 开头，消息 `etcd2 不再是支持的存储后端` - -在使用 `--storage-backend = etcd2` 升级 v1.12.x kube-apiserver 到 v1.13.x 之前，etcd v2 数据必须迁移到 v3 存储后端，并且 kube-apiserver 调用改为使用 `--storage-backend=etcd3`。 - -从 etcd2 迁移到 etcd3 的过程在很大程度上取决于部署和配置 etcd 集群的方式，以及如何部署和配置 Kubernetes 集群。 我们建议您查阅集群提供商的文档，以了解是否存在预定义的解决方案。 - -如果您的集群是通过 `kube-up.sh` 创建的并且仍然使用 etcd2 作为其存储后端，请参阅 [Kubernetes v1.12 etcd 集群升级文档](https://v1-12.docs.kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#upgrading-and-rolling-back-etcd-clusters) - - - -## 已知问题：具有安全端点的 etcd 客户端均衡器 - -在 etcd v3.3.13 或更早版本的 etcd v3 客户端有一个[严重的错误](https://github.com/kubernetes/kubernetes/issues/72102)，会影响 kube-apiserver 和 HA 部署。etcd 客户端平衡器故障转移不适用于安全端点。结果是，etcd 服务器可能会失败或短暂地与 kube-apiserver 断开连接。这会影响 kube-apiserver HA 的部署。 - -该修复程序是在 [etcd v3.4](https://github.com/etcd-io/etcd/pull/10911) 中进行的（并反向移植到 v3.3.14 或更高版本）：现在，新客户端将创建自己的凭证捆绑包，以在拨号功能中正确设置授权目标。 - -因为此修复程序要求将 gRPC 依赖升级（到 v1.23.0 ），因此，下游 Kubernetes [未反向移植 etcd 升级](https://github.com/kubernetes/kubernetes/issues/72102#issuecomment-526645978)。这意味着只能从 Kubernetes 1.16 获得 [kube-apiserver 中的 etcd 修复](https://github.com/etcd-io/etcd/pull/10911/commits/db61ee106ca9363ba3f188ecf27d1a8843da33ab)。 - -要紧急修复 Kubernetes 1.15 或更早版本的此错误，请构建一个自定义的 kube-apiserver 。 您可以使用[`vendor/google.golang.org/grpc/credentials/credentials.go`](https://github.com/kubernetes/kubernetes/blob/7b85be021cd2943167cd3d6b7020f44735d9d90b/vendor/google.golang.org/grpc/credentials/credentials.go#L135) 和 [etcd@db61ee106](https://github.com/etcd-io/etcd/pull/10911/commits/db61ee106ca9363ba3f188ecf27d1a8843da33ab) 来进行本地更改。 - -请看 ["kube-apiserver 1.13.x refuses to work when first etcd-server is not available"](https://github.com/kubernetes/kubernetes/issues/72102). - +{{< note >}} \ No newline at end of file From a71ba8d63b82fe93959a63661ac1bb20d6209efc Mon Sep 17 00:00:00 2001 From: alessandroargentieri Date: Thu, 8 Apr 2021 19:31:43 +0200 Subject: [PATCH 058/218] #20959 Localize Images page to Italian --- content/it/docs/concepts/containers/images.md | 316 ++++++++++++++++++ 1 file changed, 316 insertions(+) create mode 100644 content/it/docs/concepts/containers/images.md diff --git a/content/it/docs/concepts/containers/images.md b/content/it/docs/concepts/containers/images.md new file mode 100644 index 0000000000..43fe439ee0 --- /dev/null +++ b/content/it/docs/concepts/containers/images.md @@ -0,0 +1,316 @@ +--- +title: Immagini +content_type: concept +weight: 10 +--- + + + +L'immagine di un container rappresenta dati binari che incapsulano un'applicazione e +tutte le sue dipendenze software. Le immagini sono costituite da pacchetti software +eseguibili che possono essere avviati in modalità standalone e su cui si possono fare +ipotesi ben precise circa l'ambiente in cui vengono eseguiti. + +Tipicamente viene creata un'immagine di un'applicazione ed effettuato il _push_ +su un registry (un repository pubblico di immagini) prima di poterne fare riferimento esplicito in un +{{< glossary_tooltip text="Pod" term_id="pod" >}} + +Questa pagina va a delineare nello specifico il concetto di immagine di un container. + + + +## I nomi delle immagini + +Alle immagini dei container vengono normalmente attribuiti nomi come `pause`, `example/mycontainer`, o `kube-apiserver`. +Le immagini possono anche contenere l'hostname del registry in cui le immagini sono pubblicate; +ad esempio: `registro.fittizio.esempio/nomeimmagine`, +ed è possibile che sia incluso nel nome anche il numero della porta; ad esempio: `registro.fittizio.esempio:10443/nomeimmagine`. + +Se non si specifica l'hostname di un registry, Kubernetes assume che ci si riferisca al registry pubblico di Docker. + +Dopo la parte relativa al nome dell'immagine si può aggiungere un _tag_ (come comunemente avviene per comandi come `docker` e `podman`). +I tag permettono l'identificazione di differenti versioni della stessa serie di immagini. + +I tag delle immagini sono composti da lettere minuscole e maiuscole, numeri, underscore (`_`), +punti (`.`), e trattini (`-`). +Esistono regole aggiuntive relative a dove i caratteri separatori (`_`, `-`, and `.`) +possano essere inseriti nel tag di un'immagine. +Se non si specifica un tag, Kubernetes assume il tag `latest` che va a definire l'immagine disponibile più recente. + +{{< caution >}} +Evitate di utilizzare il tag `latest` quando si rilasciano dei container in produzione, +in quanto risulta difficile tracciare quale versione dell'immagine sia stata avviata e persino più difficile +effettuare un rollback ad una versione precente. + +Invece, meglio specificare un tag specifico come ad esempio `v1.42.0`. +{{< /caution >}} + +## Aggiornamento delle immagini + +Quando un {{< glossary_tooltip text="Deployment" term_id="deployment" >}}, +{{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}}, Pod, o qualsiasi altro +oggetto che includa un Pod template viene creato per la prima volta, la policy di default per il pull di tutti i container nel Pod +è impostata su `IfNotPresent` (se non presente) se non specificato diversamente. +Questa policy permette al +{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} di evitare di fare il pull +di un'immagine se questa è già presente. + +Se necessario, si può forzare il pull in ogni occasione in uno dei seguenti modi: + +- impostando `imagePullPolicy` (specifica per il pull delle immagini) del container su `Always` (sempre). +- omettendo `imagePullPolicy` ed usando il tag `:latest` (più recente) per l'immagine da utilizzare; + Kubernetes imposterà la policy su `Always` (sempre). +- omettendo `imagePullPolicy` ed il tag per l'immagine da utilizzare. +- abilitando l'admission controller [AlwaysPullImages](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages). + +{{< note >}} +Il valore dell'impostazione `imagePullPolicy` del container è sempre presente quando l'oggetto viene creato per la prima volta +e non viene aggiornato se il tag dell'immagine dovesse cambiare successivamente. + +Ad esempio, creando un Deployment con un'immagine il cui tag _non_ è +`:latest`, e successivamente aggiornando il tag di quell'immagine a `:latest`, il campo + `imagePullPolicy` _non_ cambierà su `Always`. +È necessario modificare manualmente la policy di pull di ogni oggetto dopo la sua creazione. +{{< /note >}} + +Quando `imagePullPolicy` è definito senza un valore specifico, esso è impostato su `Always`. + +## Multi-architecture support nelle immagini + +Oltre a fornire immagini binarie, un _container registry_ può fornire un [indice delle immagini disponibili per un container](https://github.com/opencontainers/image-spec/blob/master/image-index.md). +L'indice di un'immagine può puntare a più [file manifest](https://github.com/opencontainers/image-spec/blob/master/manifest.md) ciascuno per una versione specifica dell'architettura di un container. +L'idea è che si può avere un unico nome per una stessa immagine (ad esempio: `pause`, `example/mycontainer`, `kube-apiserver`) e permettere a diversi sistemi di recuperare l'immagine binaria corretta a seconda dell'architettura della macchina che la sta utilizzando. + + +Kubernetes stesso tipicamente nomina le immagini dei container tramite il suffisso `-$(ARCH)`. +Per la garantire la retrocompatibilità è meglio generare le vecchie immagini con dei suffissi. +L'idea è quella di generare, ad esempio, l'immagine `pause` con un manifest che include tutte le architetture supportate, +affiancata, ad esempio, da `pause-amd64` che è retrocompatibile per le vecchie configurazioni o per quei file YAML +in cui sono specificate le immagini con i suffissi. + +## Utilizzare un private registry + +I private registry possono richiedere l'utilizzo di chiavi per accedere alle immagini in essi contenute. +Le credenziali possono essere fornite in molti modi: + - configurando i nodi in modo tale da autenticarsi al private registry + - tutti i pod possono acquisire informazioni da qualsiasi private registry configurato + - è necessario che l'amministratore del cluster configuri i nodi in tal senso + - tramite pre-pulled images (immagini pre-caricate sui nodi) + - tutti i pod possono accedere alle immagini salvate sulla cache del nodo a cui si riferiscono + - è necessario effettuare l'accesso come root di sistema su ogni nodo per inserire questa impostazione + - specificando _ImagePullSecrets_ su un determinato pod + - solo i pod che forniscono le proprie chiavi hanno la possibilità di accedere al private registry + - tramite estensioni locali o specifiche di un _Vendor_ + - se si sta utilizzando una configurazione personalizzata del nodo oppure se manualmente, o tramite il _cloud provider_, + si implementa un meccanismo di autenticazione del nodo presso il _container registry_. + +Di seguito la spiegazione dettagliata di queste opzioni. + +### Configurazione dei nodi per l'autenticazione ad un private registry + +Se si sta utilizzando Docker sui nodi, si può configurare il _Docker container runtime_ +per autenticare il nodo presso un private container registry. + +Questo è un approccio possibile se si ha il controllo sulle configurazioni del nodo. + +{{< note >}} +Kubernetes di default supporta solo le sezioni `auths` e `HttpHeaders` nelle configurazioni relative a Docker. +Eventuali _helper_ per le credenziali di Docker (`credHelpers` o `credsStore`) non sono supportati. +{{< /note >}} + + +Docker salva le chiavi per i registri privati in `$HOME/.dockercfg` oppure nel file `$HOME/.docker/config.json`. +Inserendo lo stesso file nella lista seguente, kubelet lo utilizzerà per recuperare le credenziali quando deve fare il _pull_ delle immagini. + +* `{--root-dir:-/var/lib/kubelet}/config.json` +* `{cwd of kubelet}/config.json` +* `${HOME}/.docker/config.json` +* `/.docker/config.json` +* `{--root-dir:-/var/lib/kubelet}/.dockercfg` +* `{cwd of kubelet}/.dockercfg` +* `${HOME}/.dockercfg` +* `/.dockercfg` + +{{< note >}} +Potrebbe essere necessario impostare `HOME=/root` esplicitamente come variabile d'ambiente del processo _kubelet_. +{{< /note >}} + +Di seguito i passi consigliati per configurare l'utilizzo di un private registry da parte dei nodi del _cluster_. +In questo esempio, eseguire i seguenti comandi sul proprio desktop/laptop: + + 1. Esegui `docker login [server]` per ogni _set_ di credenziali che vuoi utilizzare. Questo comando aggiornerà `$HOME/.docker/config.json` sul tuo PC. + 1. Controlla il file `$HOME/.docker/config.json` in un editor di testo per assicurarti che contenga le credenziali che tu voglia utilizzare. + 1. Recupera la lista dei tuoi nodi; ad esempio: + - se vuoi utilizzare i nomi: `nodes=$( kubectl get nodes -o jsonpath='{range.items[*].metadata}{.name} {end}' )` + - se vuoi recuperare gli indirizzi IP: `nodes=$( kubectl get nodes -o jsonpath='{range .items[*].status.addresses[?(@.type=="ExternalIP")]}{.address} {end}' )` + 1. Copia il tuo file locale `.docker/config.json` in uno dei path sopra riportati nella lista di ricerca. + - ad esempio, per testare il tutto: `for n in $nodes; do scp ~/.docker/config.json root@"$n":/var/lib/kubelet/config.json; done` + +{{< note >}} +Per i cluster di produzione, utilizza un configuration management tool per poter applicare le impostazioni su tutti i nodi laddove necessario. +{{< /note >}} + +Puoi fare una verifica creando un Pod che faccia uso di un'immagine privata; ad esempio: + +```shell +kubectl apply -f - <}} +Questo approccio è possibile se si ha il controllo sulla configurazione del nodo. +Non funzionerà qualora il cloud provider gestisca i nodi e li sostituisca automaticamente. +{{< /note >}} + +Kubelet di default prova a fare il pull di ogni immagine dal registry specificato. +Tuttavia, qualora la proprietà `imagePullPolicy` (specifica di pull dell'immagine) del container sia impostata su `IfNotPresent` (vale a dire, se non è già presente) oppure su `Never` (mai), +allora l'immagine locale è utilizzata (in via preferenziale o esclusiva, rispettivamente). + +Se si vuole fare affidamento a immagini pre-scaricate per non dover incorrere in una fase di autenticazione presso il registry, +bisogna assicurarsi che tutti i nodi nel cluster abbiano scaricato le stesse versioni delle immagini. + +Questa procedura può essere utilizzata per accelerare il processo di creazione delle istanze o come alternativa all'autenticazione presso un private registry. + +Tutti i pod avranno accesso in lettura a qualsiasi immagine pre-scaricata. + +### Specificare la proprietà imagePullSecrets su un Pod + +{{< note >}} +Questo approccio è quello consigliato per l'avvio di container a partire da immagini presenti in registri privati. +{{< /note >}} + +Kubernetes da la possibilità di specificare le chiavi del _container registry_ su un Pod. + +#### Creare un Secret tramite Docker config + +Esegui il comando seguente, sostituendo i valori riportati in maiuscolo con quelli corretti: + +```shell +kubectl create secret docker-registry --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL +``` + +Se possiedi il file delle credenziali per Docker, anziché utilizzare il comando quì sopra +puoi importare il file di credenziali come un Kubernetes +{{< glossary_tooltip text="Secrets" term_id="secret" >}}. +[Creare un Secret a partire da credenziali Docker](/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) fornisce la spiegazione dettagliata su come fare. + +Ciò è particolarmente utile se si utilizzano più _container registry_ privati, +in quanto il comando `kubectl create secret docker-registry` genera un Secret che +funziona con un solo private registry. + +{{< note >}} +I Pod possono fare riferimento ai Secret per il pull delle immagini soltanto nel proprio _namespace_, +quindi questo procedimento deve essere svolto per ogni _namespace_. +{{< /note >}} + +#### Fare riferimento ad imagePullSecrets in un Pod + +È possibile creare pod che referenzino quel Secret aggiungendo la sezione `imagePullSecrets` alla definizione del Pod. + +Ad esempio: + +```shell +cat < pod.yaml +apiVersion: v1 +kind: Pod +metadata: + name: foo + namespace: awesomeapps +spec: + containers: + - name: foo + image: janedoe/awesomeapp:v1 + imagePullSecrets: + - name: myregistrykey +EOF + +cat <> ./kustomization.yaml +resources: +- pod.yaml +EOF +``` + +Questo deve esser fatto per ogni Pod che utilizzi un private registry. + +Comunque, le impostazioni relative a questo campo possono essere automatizzate inserendo la sezione _imagePullSecrets_ +nella definizione della risorsa [ServiceAccount](/docs/tasks/configure-pod-container/configure-service-account/). + +Visitare la pagina [Aggiungere ImagePullSecrets ad un Service Account](/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account) per istruzioni più dettagliate. + +Puoi utilizzarlo in congiunzione al file `.docker/config.json` configurato per ogni nodo. In questo caso, si applicherà un _merge_ delle credenziali. + +## Casi d'uso + +Ci sono varie soluzioni per configurare i private registry. Di seguito, alcuni casi d'uso comuni e le soluzioni suggerite. + +1. Cluster in cui sono utilizzate soltanto immagini non proprietarie (ovvero _open-source_). In questo caso non sussiste il bisogno di nascondere le immagini. + - Utilizza immagini pubbliche da Docker hub. + - Nessuna configurazione richiesta. + - Alcuni _cloud provider_ mettono in _cache_ o effettuano il _mirror_ di immagini pubbliche, il che migliora la disponibilità delle immagini e ne riduce il tempo di _pull_. +1. Cluster con container avviati a partire da immagini proprietarie che dovrebbero essere nascoste a chi è esterno all'organizzazione, ma + visibili a tutti gli utenti abilitati nel cluster. + - Utilizza un private [Docker registry](https://docs.docker.com/registry/). + - Esso può essere ospitato da [Docker Hub](https://hub.docker.com/signup), o da qualche altra piattaforma. + - Configura manualmente il file .docker/config.json su ogni nodo come descritto sopra. + - Oppure, avvia un private registry dietro il tuo firewall con accesso in lettura libero. + - Non è necessaria alcuna configurazione di Kubernetes. + - Utilizza un servizio di _container registry_ che controlli l'accesso alle immagini + - Esso funzionerà meglio con una configurazione del cluster basata su _autoscaling_ che con una configurazione manuale del nodo. + - Oppure, su un cluster dove la modifica delle configurazioni del nodo non è conveniente, utilizza `imagePullSecrets`. +1. Cluster con immagini proprietarie, alcune delle quali richiedono un controllo sugli accessi. + - Assicurati che l'_admission controller_ [AlwaysPullImages](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages) sia attivo. Altrimenti, tutti i Pod potenzialmente possono avere accesso a tutte le immagini. + - Sposta i dati sensibili un un _Secret_, invece di inserirli in un'immagine. +1. Un cluster multi-tenant dove ogni tenant necessiti di un private registry. + - Assicurati che l'_admission controller_ [AlwaysPullImages](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages) sia attivo. Altrimenti, tutti i Pod di tutti i tenant potrebbero potenzialmente avere accesso a tutte le immagini. + - Avvia un private registry che richieda un'autorizzazione all'accesso. + - Genera delle credenziali di registry per ogni tenant, inseriscile in dei _Secret_, e popola i _Secret_ per ogni _namespace_ relativo ad ognuno dei tenant. + - Il singolo tenant aggiunge così quel _Secret_ all'impostazione _imagePullSecrets_ di ogni _namespace_. + + +Se si ha la necessità di accedere a più registri, si può generare un _Secret_ per ognuno di essi. +Kubelet farà il _merge_ di ogni `imagePullSecrets` in un singolo file virtuale `.docker/config.json`. + +## {{% heading "whatsnext" %}} + +* Leggi [OCI Image Manifest Specification](https://github.com/opencontainers/image-spec/blob/master/manifest.md) \ No newline at end of file From a5c84d959fd87bb9984ec98e53cd994a7ff25592 Mon Sep 17 00:00:00 2001 From: Rey Lejano Date: Thu, 8 Apr 2021 12:39:44 -0700 Subject: [PATCH 059/218] Update release notes for 1.21 release --- content/en/docs/setup/release/notes.md | 3414 +++++++++++------------- 1 file changed, 1494 insertions(+), 1920 deletions(-) diff --git a/content/en/docs/setup/release/notes.md b/content/en/docs/setup/release/notes.md index 54146007a0..89dc5694c0 100644 --- a/content/en/docs/setup/release/notes.md +++ b/content/en/docs/setup/release/notes.md @@ -1,5 +1,5 @@ --- -title: v1.20 Release Notes +title: v1.21 Release Notes weight: 10 card: name: release-notes @@ -13,1592 +13,760 @@ card: -# v1.20.0 +# v1.21.0 [Documentation](https://docs.k8s.io) -## Downloads for v1.20.0 +## Downloads for v1.21.0 filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes.tar.gz) | `ebfe49552bbda02807034488967b3b62bf9e3e507d56245e298c4c19090387136572c1fca789e772a5e8a19535531d01dcedb61980e42ca7b0461d3864df2c14` -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-src.tar.gz) | `bcbd67ed0bb77840828c08c6118ad0c9bf2bcda16763afaafd8731fd6ce735be654feef61e554bcc34c77c65b02a25dae565adc5e1dc49a2daaa0d115bf1efe6` +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes.tar.gz) | `19bb76a3fa5ce4b9f043b2a3a77c32365ab1fcb902d8dd6678427fb8be8f49f64a5a03dc46aaef9c7dadee05501cf83412eda46f0edacbb8fc1ed0bf5fb79142` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-src.tar.gz) | `f942e6d6c10007a6e9ce21e94df597015ae646a7bc3e515caf1a3b79f1354efb9aff59c40f2553a8e3d43fe4a01742241f5af18b69666244906ed11a22e3bc49` ### Client Binaries filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-darwin-amd64.tar.gz) | `3609f6483f4244676162232b3294d7a2dc40ae5bdd86a842a05aa768f5223b8f50e1d6420fd8afb2d0ce19de06e1d38e5e5b10154ba0cb71a74233e6dc94d5a0` -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-386.tar.gz) | `e06c08016a08137d39804383fdc33a40bb2567aa77d88a5c3fd5b9d93f5b581c635b2c4faaa718ed3bb2d120cb14fe91649ed4469ba72c3a3dda1e343db545ed` -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-amd64.tar.gz) | `081472833601aa4fa78e79239f67833aa4efcb4efe714426cd01d4ddf6f36fbf304ef7e1f5373bff0fdff44a845f7560165c093c108bd359b5ab4189f36b1f2f` -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-arm.tar.gz) | `037f84a2f29fe62d266cab38ac5600d058cce12cbc4851bcf062fafba796c1fbe23a0c2939cd15784854ca7cd92383e5b96a11474fc71fb614b47dbf98a477d9` -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-arm64.tar.gz) | `275727e1796791ca3cbe52aaa713a2660404eab6209466fdc1cfa8559c9b361fe55c64c6bcecbdeba536b6d56213ddf726e58adc60f959b6f77e4017834c5622` -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-ppc64le.tar.gz) | `7a9965293029e9fcdb2b7387467f022d2026953b8461e6c84182abf35c28b7822d2389a6d8e4d8e532d2ea5d5d67c6fee5fb6c351363cb44c599dc8800649b04` -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-s390x.tar.gz) | `85fc449ce1980f5f030cc32e8c8e2198c1cc91a448e04b15d27debc3ca56aa85d283f44b4f4e5fed26ac96904cc12808fa3e9af3d8bf823fc928befb9950d6f5` -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-windows-386.tar.gz) | `4c0a27dba1077aaee943e0eb7a787239dd697e1d968e78d1933c1e60b02d5d233d58541d5beec59807a4ffe3351d5152359e11da120bf64cacb3ee29fbc242e6` -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-windows-amd64.tar.gz) | `29336faf7c596539b8329afbbdceeddc843162501de4afee44a40616278fa1f284d8fc48c241fc7d52c65dab70f76280cc33cec419c8c5dbc2625d9175534af8` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-darwin-amd64.tar.gz) | `be9d1440e418e5253fb8a3d8aba705ca8160746a9bd17325ad626a986b6da9f733af864155a651a32b7bca94b533b8d596005ddbe5248bdeea85db47a1b957ed` +[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-darwin-arm64.tar.gz) | `eed0ddc81d104bb2d41ace13f737c490423d5df4ebddc7376e45c18ed66af35933c9376b912c1c3da105945b04056f6ca0870c156bee8a307cf4189ca5eb1dd1` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-386.tar.gz) | `8a2f30c4434199762f2a96141dab4241c1cce2711bea9ea39cc63c2c5e7d31719ed7f076efac1931604e3a94578d3bbf0cfa454965708c96f3cfb91789868746` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-amd64.tar.gz) | `cd3cfa645fa31de3716f1f63506e31b73d2aa8d37bb558bb3b3e8c151f35b3d74d44e03cbd05be67e380f9a5d015aba460222afdac6677815cd99a85c2325cf0` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-arm.tar.gz) | `936042aa11cea0f6dfd2c30fc5dbe655420b34799bede036b1299a92d6831f589ca10290b73b9c9741560b603ae31e450ad024e273f2b4df5354bfac272691d8` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-arm64.tar.gz) | `42beb75364d7bf4bf526804b8a35bd0ab3e124b712e9d1f45c1b914e6be0166619b30695feb24b3eecef134991dacb9ab3597e788bd9e45cf35addddf20dd7f6` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-ppc64le.tar.gz) | `4baba2ed7046b28370eccc22e2378ae79e3ce58220d6f4f1b6791e8233bec8379e30200bb20b971456b83f2b791ea166fdfcf1ea56908bc1eea03590c0eda468` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-linux-s390x.tar.gz) | `37fa0c4d703aef09ce68c10ef3e7362b0313c8f251ce38eea579cd18fae4023d3d2b70e0f31577cabe6958ab9cfc30e98d25a7c64e69048b423057c3cf728339` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-windows-386.tar.gz) | `6900db36c1e3340edfd6dfd8d720575a904c932d39a8a7fa36401595e971a0235bd42111dbcc1cbb77e7374e47f1380a68c637997c18f96a0d9cdc9f3714c4c9` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-client-windows-amd64.tar.gz) | `90de67f6f79fc63bcfdf35066e3d84501cc85433265ffad36fd1a7a428a31b446249f0644a1e97495ea8b2a08e6944df6ef30363003750339edaa2aceffe937c` ### Server Binaries filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-amd64.tar.gz) | `fb56486a55dbf7dbacb53b1aaa690bae18d33d244c72a1e2dc95fb0fcce45108c44ba79f8fa04f12383801c46813dc33d2d0eb2203035cdce1078871595e446e` -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-arm.tar.gz) | `735ed9993071fe35b292bf06930ee3c0f889e3c7edb983195b1c8e4d7113047c12c0f8281fe71879fc2fcd871e1ee587f03b695a03c8512c873abad444997a19` -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-arm64.tar.gz) | `ffab155531d5a9b82487ee1abf4f6ef49626ea58b2de340656a762e46cf3e0f470bdbe7821210901fe1114224957c44c1d9cc1e32efb5ee24e51fe63990785b2` -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-ppc64le.tar.gz) | `9d5730d35c4ddfb4c5483173629fe55df35d1e535d96f02459468220ac2c97dc01b995f577432a6e4d1548b6edbfdc90828dc9c1f7cf7464481af6ae10aaf118` -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-s390x.tar.gz) | `6e4c165306940e8b99dd6e590f8542e31aed23d2c7a6808af0357fa425cec1a57016dd66169cf2a95f8eb8ef70e1f29e2d500533aae889e2e3d9290d04ab8721` +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-amd64.tar.gz) | `3941dcc2309ac19ec185603a79f5a086d8a198f98c04efa23f15a177e5e1f34946ea9392ba9f5d24d0d727839438f067fef1001fc6e88b27b8b01e35bbd962ca` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-arm.tar.gz) | `6507abf6c2ec2b336901dc23269f6c577ec0049b8bad3c9dd6ad63f21aa10f09bfbbfa6e064c2466d250411d3e10f8672791a9e10942e38de7bfbaf7a8bcc9da` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-arm64.tar.gz) | `5abe76f867ca6865344e957bf166b81766c049ec4eb183a8a5580c22a7f8474db1edf90fd901a5833e56128b6825811653a1d27f72fd34ce5b1287a8c10da05c` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-ppc64le.tar.gz) | `62507b182ca25396a285d91241536860e58f54fac937e97cbdf91948c83bb41be97d33277400489bf50e85164d560205540b76e94e5d519892312bdc63df1067` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-server-linux-s390x.tar.gz) | `04f2a1f7d1388e4a7d7d9f597f872a3da36f26839cfed16aad6df07021c03f4dca1df06b19cfda56df09d1c2d9a13ebd0af40ca1b9b6aecfaf427ab7712d88f3` ### Node Binaries filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-amd64.tar.gz) | `3e6c90561dd1c27fa1dff6953c503251c36001f7e0f8eff3ec918c74ae2d9aa25917d8ac87d5b4224b8229f620b1830442e6dce3b2a497043f8497eee3705696` -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-arm.tar.gz) | `26db385d9ae9a97a1051a638e7e3de22c4bbff389d5a419fe40d5893f9e4fa85c8b60a2bd1d370fd381b60c3ca33c5d72d4767c90898caa9dbd4df6bd116a247` -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-arm64.tar.gz) | `5b8b63f617e248432b7eb913285a8ef8ba028255216332c05db949666c3f9e9cb9f4c393bbd68d00369bda77abf9bfa2da254a5c9fe0d79ffdad855a77a9d8ed` -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-ppc64le.tar.gz) | `60da7715996b4865e390640525d6e98593ba3cd45c6caeea763aa5355a7f989926da54f58cc5f657f614c8134f97cd3894b899f8b467d100dca48bc22dd4ff63` -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-s390x.tar.gz) | `9407dc55412bd04633f84fcefe3a1074f3eaa772a7cb9302242b8768d6189b75d37677a959f91130e8ad9dc590f9ba8408ba6700a0ceff6827315226dd5ee1e6` -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-windows-amd64.tar.gz) | `9d4261af343cc330e6359582f80dbd6efb57d41f882747a94bbf47b4f93292d43dd19a86214d4944d268941622dfbc96847585e6fec15fddc4dbd93d17015fa8` +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-amd64.tar.gz) | `c1831c708109c31b3878e5a9327ea4b9e546504d0b6b00f3d43db78b5dd7d5114d32ac24a9a505f9cadbe61521f0419933348d2cd309ed8cfe3987d9ca8a7e2c` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-arm.tar.gz) | `b68dd5bcfc7f9ce2781952df40c8c3a64c29701beff6ac22f042d6f31d4de220e9200b7e8272ddf608114327770acdaf3cb9a34a0a5206e784bda717ea080e0f` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-arm64.tar.gz) | `7fa84fc500c28774ed25ca34b6f7b208a2bea29d6e8379f84b9f57bd024aa8fe574418cee7ee26edd55310716d43d65ae7b9cbe11e40c995fe2eac7f66bdb423` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-ppc64le.tar.gz) | `a4278b3f8e458e9581e01f0c5ba8443303c987988ee136075a8f2f25515d70ca549fbd2e4d10eefca816c75c381d62d71494bd70c47034ab47f8315bbef4ae37` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-linux-s390x.tar.gz) | `8de2bc6f22f232ff534b45012986eac23893581ccb6c45bd637e40dbe808ce31d5a92375c00dc578bdbadec342b6e5b70c1b9f3d3a7bb26ccfde97d71f9bf84a` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0/kubernetes-node-windows-amd64.tar.gz) | `b82e94663d330cff7a117f99a7544f27d0bc92b36b5a283b3c23725d5b33e6f15e0ebf784627638f22f2d58c58c0c2b618ddfd226a64ae779693a0861475d355` -## Changelog since v1.19.0 +## Changelog since v1.20.0 + +# Release notes for v1.21.0-rc.0 + +[Documentation](https://docs.k8s.io/docs/home) + +# Changelog since v1.20.0 ## What's New (Major Themes) -### Dockershim deprecation +### Deprecation of PodSecurityPolicy -Docker as an underlying runtime is being deprecated. Docker-produced images will continue to work in your cluster with all runtimes, as they always have. -The Kubernetes community [has written a blog post about this in detail](https://blog.k8s.io/2020/12/02/dont-panic-kubernetes-and-docker/) with [a dedicated FAQ page for it](https://blog.k8s.io/2020/12/02/dockershim-faq/). +PSP as an admission controller resource is being deprecated. Deployed PodSecurityPolicy's will keep working until version 1.25, their target removal from the codebase. A new feature, with a working title of "PSP replacement policy", is being developed in [KEP-2579](https://features.k8s.io/2579). To learn more, read [PodSecurityPolicy Deprecation: Past, Present, and Future](https://blog.k8s.io/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). -### External credential provider for client-go +### Kubernetes API Reference Documentation -The client-go credential plugins can now be passed in the current cluster information via the `KUBERNETES_EXEC_INFO` environment variable. Learn more about this on [client-go credential plugins documentation](https://docs.k8s.io/reference/access-authn-authz/authentication/#client-go-credential-plugins/). +The API reference is now generated with [`gen-resourcesdocs`](https://github.com/kubernetes-sigs/reference-docs/tree/c96658d89fb21037b7d00d27e6dbbe6b32375837/gen-resourcesdocs) and it is moving to [Kubernetes API](https://docs.k8s.io/reference/kubernetes-api/) -### CronJob controller v2 is available through feature gate +### Kustomize Updates in Kubectl -An alternative implementation of `CronJob` controller is now available as an alpha feature in this release, which has experimental performance improvement by using informers instead of polling. While this will be the default behavior in the future, you can [try them in this release through a feature gate](https://docs.k8s.io/concepts/workloads/controllers/cron-jobs/). +[Kustomize](https://github.com/kubernetes-sigs/kustomize) version in kubectl had a jump from v2.0.3 to [v4.0.5](https://github.com/kubernetes/kubernetes/pull/98946). Kustomize is now treated as a library and future updates will be less sporadic. -### PID Limits graduates to General Availability +### Default Container Labels -PID Limits features are now generally available on both `SupportNodePidsLimit` (node-to-pod PID isolation) and `SupportPodPidsLimit` (ability to limit PIDs per pod), after being enabled-by-default in beta stage for a year. +Pod with multiple containers can use `kubectl.kubernetes.io/default-container` label to have a container preselected for kubectl commands. More can be read in [KEP-2227](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/2227-kubectl-default-container/README.md). -### API Priority and Fairness graduates to Beta +### Immutable Secrets and ConfigMaps -Initially introduced in 1.18, Kubernetes 1.20 now enables API Priority and Fairness (APF) by default. This allows `kube-apiserver` to [categorize incoming requests by priority levels](https://docs.k8s.io/concepts/cluster-administration/flow-control/). +Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load. -### IPv4/IPv6 run +### Structured Logging in Kubelet -IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services, based on user and community feedback. If your cluster has dual-stack enabled, you can create Services which can use IPv4, IPv6, or both, and you can change this setting for existing Services. Details are available in updated [IPv4/IPv6 dual-stack docs](https://docs.k8s.io/concepts/services-networking/dual-stack/), which cover the nuanced array of options. +Kubelet has adopted structured logging, thanks to community effort in accomplishing this within the release timeline. Structured logging in the project remains an ongoing effort -- for folks interested in participating, [keep an eye / chime in to the mailing list discussion](https://groups.google.com/g/kubernetes-dev/c/y4WIw-ntUR8). -We expect this implementation to progress from alpha to beta and GA in coming releases, so we’re eager to have you comment about your dual-stack experiences in [#k8s-dual-stack](https://kubernetes.slack.com/messages/k8s-dual-stack) or in [enhancements #563](https://features.k8s.io/563). +### Storage Capacity Tracking -### go1.15.5 +Traditionally, the Kubernetes scheduler was based on the assumptions that additional persistent storage is available everywhere in the cluster and has infinite capacity. Topology constraints addressed the first point, but up to now pod scheduling was still done without considering that the remaining storage capacity may not be enough to start a new pod. [Storage capacity tracking](https://docs.k8s.io/concepts/storage/storage-capacity/) addresses that by adding an API for a CSI driver to report storage capacity and uses that information in the Kubernetes scheduler when choosing a node for a pod. This feature serves as a stepping stone for supporting dynamic provisioning for local volumes and other volume types that are more capacity constrained. -go1.15.5 has been integrated to Kubernetes project as of this release, [including other infrastructure related updates on this effort](https://github.com/kubernetes/kubernetes/pull/95776). +### Generic Ephemeral Volumes -### CSI Volume Snapshot graduates to General Availability +[Generic ephermeral volumes](https://docs.k8s.io/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. It can be used to provide scratch storage that is different from the root disk, for example persistent memory, or a separate local disk on that node. All StorageClass parameters for volume provisioning are supported. All features supported with PersistentVolumeClaims are supported, such as storage capacity tracking, snapshots and restore, and volume resizing. -CSI Volume Snapshot moves to GA in the 1.20 release. This feature provides a standard way to trigger volume snapshot operations in Kubernetes and allows Kubernetes users to incorporate snapshot operations in a portable manner on any Kubernetes environment regardless of supporting underlying storage providers. -Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise grade, storage administration features for Kubernetes: including application or cluster level backup solutions. -Note that snapshot support will require Kubernetes distributors to bundle the Snapshot controller, Snapshot CRDs, and validation webhook. In addition, a CSI driver supporting the snapshot functionality must also be deployed on the cluster. +### CSI Service Account Token -### Non-recursive Volume Ownership (FSGroup) graduates to Beta +CSI Service Account Token feature moves to Beta in 1.21. This feature improves the security posture and allows CSI drivers to receive pods' [bound service account tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md). This feature also provides a knob to re-publish volumes so that short-lived volumes can be refreshed. -By default, the `fsgroup` setting, if specified, recursively updates permissions for every file in a volume on every mount. This can make mount, and pod startup, very slow if the volume has many files. -This setting enables a pod to specify a `PodFSGroupChangePolicy` that indicates that volume ownership and permissions will be changed only when permission and ownership of the root directory does not match with expected permissions on the volume. +### CSI Health Monitoring -### CSIDriver policy for FSGroup graduates to Beta - -The FSGroup's CSIDriver Policy is now beta in 1.20. This allows CSIDrivers to explicitly indicate if they want Kubernetes to manage permissions and ownership for their volumes via `fsgroup`. - -### Security Improvements for CSI Drivers (Alpha) - -In 1.20, we introduce a new alpha feature `CSIServiceAccountToken`. This feature allows CSI drivers to impersonate the pods that they mount the volumes for. This improves the security posture in the mounting process where the volumes are ACL’ed on the pods’ service account without handing out unnecessary permissions to the CSI drivers’ service account. This feature is especially important for secret-handling CSI drivers, such as the secrets-store-csi-driver. Since these tokens can be rotated and short-lived, this feature also provides a knob for CSI drivers to receive `NodePublishVolume` RPC calls periodically with the new token. This knob is also useful when volumes are short-lived, e.g. certificates. - -### Introducing Graceful Node Shutdown (Alpha) - -The `GracefulNodeShutdown` feature is now in Alpha. This allows kubelet to be aware of node system shutdowns, enabling graceful termination of pods during a system shutdown. This feature can be [enabled through feature gate](https://docs.k8s.io/concepts/architecture/nodes/#graceful-node-shutdown). - -### Runtime log sanitation - -Logs can now be configured to use runtime protection from leaking sensitive data. [Details for this experimental feature is available in documentation](https://docs.k8s.io/concepts/cluster-administration/system-logs/#log-sanitization). - -### Pod resource metrics - -On-demand metrics calculation is now available through `/metrics/resources`. [When enabled]( -https://docs.k8s.io/concepts/cluster-administration/system-metrics#kube-scheduler-metrics), the endpoint will report the requested resources and the desired limits of all running pods. - -### Introducing `RootCAConfigMap` - -`RootCAConfigMap` graduates to Beta, seperating from `BoundServiceAccountTokenVolume`. The `kube-root-ca.crt` ConfigMap is now available to every namespace, by default. It contains the Certificate Authority bundle for verify kube-apiserver connections. - -### `kubectl debug` graduates to Beta - -`kubectl alpha debug` graduates from alpha to beta in 1.20, becoming `kubectl debug`. -`kubectl debug` provides support for common debugging workflows directly from kubectl. Troubleshooting scenarios supported in this release of `kubectl` include: -Troubleshoot workloads that crash on startup by creating a copy of the pod that uses a different container image or command. -Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the pod or using an ephemeral container. (Ephemeral containers are an alpha feature that are not enabled by default.) -Troubleshoot on a node by creating a container running in the host namespaces and with access to the host’s filesystem. -Note that as a new builtin command, `kubectl debug` takes priority over any `kubectl` plugin named “debug”. You will need to rename the affected plugin. -Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug` instead of `kubectl alpha debug`! -For more information about kubectl debug, see Debugging Running Pods on the Kubernetes website, kubectl help debug, or reach out to SIG CLI by visiting #sig-cli or commenting on [enhancement #1441](https://features.k8s.io/1441). - -### Removing deprecated flags in kubeadm - -`kubeadm` applies a number of deprecations and removals of deprecated features in this release. More details are available in the Urgent Upgrade Notes and Kind / Deprecation sections. - -### Pod Hostname as FQDN graduates to Beta - -Previously introduced in 1.19 behind a feature gate, `SetHostnameAsFQDN` is now enabled by default. More details on this behavior is available in [documentation for DNS for Services and Pods](https://docs.k8s.io/concepts/services-networking/dns-pod-service/#pod-sethostnameasfqdn-field) - -### `TokenRequest` / `TokenRequestProjection` graduates to General Availability - -Service account tokens bound to pod is now a stable feature. The feature gates will be removed in 1.21 release. For more information, refer to notes below on the changelogs. - -### RuntimeClass feature graduates to General Availability. - -The `node.k8s.io` API groups are promoted from `v1beta1` to `v1`. `v1beta1` is now deprecated and will be removed in a future release, please start using `v1`. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] - -### Cloud Controller Manager now exclusively shipped by Cloud Provider - -Kubernetes will no longer ship an instance of the Cloud Controller Manager binary. Each Cloud Provider is expected to ship their own instance of this binary. Details for a Cloud Provider to create an instance of such a binary can be found under [here](https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/cloud-provider/sample). Anyone with questions on building a Cloud Controller Manager should reach out to SIG Cloud Provider. Questions about the Cloud Controller Manager on a Managed Kubernetes solution should go to the relevant Cloud Provider. Questions about the Cloud Controller Manager on a non managed solution can be brought up with SIG Cloud Provider. +The CSI health monitoring feature is being released as a second Alpha in Kubernetes 1.21. This feature enables CSI Drivers to share abnormal volume conditions from the underlying storage systems with Kubernetes so that they can be reported as events on PVCs or Pods. This feature serves as a stepping stone towards programmatic detection and resolution of individual volume health issues by Kubernetes. ## Known Issues -### Summary API in kubelet doesn't have accelerator metrics -Currently, cadvisor_stats_provider provides AcceleratorStats but cri_stats_provider does not. As a result, when using cri_stats_provider, kubelet's Summary API does not have accelerator metrics. [There is an open work in progress to fix this](https://github.com/kubernetes/kubernetes/pull/96873). +### `TopologyAwareHints` feature falls back to default behavior + +The feature gate currently falls back to the default behavior in most cases. Enabling the feature gate will add hints to `EndpointSlices`, but functional differences are only observed in non-dual stack kube-proxy implementation. [The fix will be available in coming releases](https://github.com/kubernetes/kubernetes/pull/100804). ## Urgent Upgrade Notes ### (No, really, you MUST read this before you upgrade) -- A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is `1s` which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts. See [configure probe](https://docs.k8s.io/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes) section of the documentation for more details. - - - This change in behavior may be unexpected for some clusters and can be disabled by turning off the `ExecProbeTimeout` feature gate. This gate will be locked and removed in future releases so that exec probe timeouts are always respected. ([#94115](https://github.com/kubernetes/kubernetes/pull/94115), [@andrewsykim](https://github.com/andrewsykim)) [SIG Node and Testing] -- RuntimeClass feature graduates to General Availability. Promote `node.k8s.io` API groups from `v1beta1` to `v1`. `v1beta1` is now deprecated and will be removed in a future release, please start using `v1`. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] -- API priority and fairness graduated to beta. 1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. ([#96527](https://github.com/kubernetes/kubernetes/pull/96527), [@adtac](https://github.com/adtac)) [SIG API Machinery and Testing] -- For CSI drivers, kubelet no longer creates the target_path for NodePublishVolume in accordance with the CSI spec. Kubelet also no longer checks if staging and target paths are mounts or corrupted. CSI drivers need to be idempotent and do any necessary mount verification. ([#88759](https://github.com/kubernetes/kubernetes/pull/88759), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] -- Kubeadm: http://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md ([#95382](https://github.com/kubernetes/kubernetes/pull/95382), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - - The label applied to control-plane nodes "node-role.kubernetes.io/master" is now deprecated and will be removed in a future release after a GA deprecation period. - - Introduce a new label "node-role.kubernetes.io/control-plane" that will be applied in parallel to "node-role.kubernetes.io/master" until the removal of the "node-role.kubernetes.io/master" label. - - Make "kubeadm upgrade apply" add the "node-role.kubernetes.io/control-plane" label on existing nodes that only have the "node-role.kubernetes.io/master" label during upgrade. - - Please adapt your tooling built on top of kubeadm to use the "node-role.kubernetes.io/control-plane" label. - - The taint applied to control-plane nodes "node-role.kubernetes.io/master:NoSchedule" is now deprecated and will be removed in a future release after a GA deprecation period. - - Apply toleration for a new, future taint "node-role.kubernetes.io/control-plane:NoSchedule" to the kubeadm CoreDNS / kube-dns managed manifests. Note that this taint is not yet applied to kubeadm control-plane nodes. - - Please adapt your workloads to tolerate the same future taint preemptively. - -- Kubeadm: improve the validation of serviceSubnet and podSubnet. - ServiceSubnet has to be limited in size, due to implementation details, and the mask can not allocate more than 20 bits. - PodSubnet validates against the corresponding cluster "--node-cidr-mask-size" of the kube-controller-manager, it fail if the values are not compatible. - kubeadm no longer sets the node-mask automatically on IPv6 deployments, you must check that your IPv6 service subnet mask is compatible with the default node mask /64 or set it accordenly. - Previously, for IPv6, if the podSubnet had a mask lower than /112, kubeadm calculated a node-mask to be multiple of eight and splitting the available bits to maximise the number used for nodes. ([#95723](https://github.com/kubernetes/kubernetes/pull/95723), [@aojea](https://github.com/aojea)) [SIG Cluster Lifecycle] -- The deprecated flag --experimental-kustomize is now removed from kubeadm commands. Use --experimental-patches instead, which was introduced in 1.19. Migration information available in --help description for --experimental-patches. ([#94871](https://github.com/kubernetes/kubernetes/pull/94871), [@neolit123](https://github.com/neolit123)) -- Windows hyper-v container featuregate is deprecated in 1.20 and will be removed in 1.21 ([#95505](https://github.com/kubernetes/kubernetes/pull/95505), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] -- The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags `--address` and `--insecure-bind-address` have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags `--port` and `--insecure-port` may only be set to 0 and will be removed in v1.24. ([#95856](https://github.com/kubernetes/kubernetes/pull/95856), [@knight42](https://github.com/knight42), [SIG API Machinery, Node, Testing]) -- Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. - It changes the dual-stack API wrt Service from a single ipFamily field to 3 - fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), - ipFamilies (a list of families assigned), and clusterIPs (inclusive of - clusterIP). Most users do not need to set anything at all, defaulting will - handle it for them. Services are single-stack unless the user asks for - dual-stack. This is all gated by the "IPv6DualStack" feature gate. ([#91824](https://github.com/kubernetes/kubernetes/pull/91824), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] -- `TokenRequest` and `TokenRequestProjection` are now GA features. The following flags are required by the API server: - - `--service-account-issuer`, should be set to a URL identifying the API server that will be stable over the cluster lifetime. - - `--service-account-key-file`, set to one or more files containing one or more public keys used to verify tokens. - - `--service-account-signing-key-file`, set to a file containing a private key to use to sign service account tokens. Can be the same file given to `kube-controller-manager` with `--service-account-private-key-file`. ([#95896](https://github.com/kubernetes/kubernetes/pull/95896), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle] -- kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags: - - apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration. - - cluster-name: use clusterName from ClusterConfiguration - - cert-dir: use certificatesDir from ClusterConfiguration ([#94879](https://github.com/kubernetes/kubernetes/pull/94879), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] -- Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of `OwnerRefInvalidNamespace` are recorded when namespace mismatches between child and owner objects are detected. The [kubectl-check-ownerreferences](https://github.com/kubernetes-sigs/kubectl-check-ownerreferences) tool can be run prior to upgrading to locate existing objects with invalid ownerReferences. - - A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted. - - A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. ([#92743](https://github.com/kubernetes/kubernetes/pull/92743), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Testing] - - +- Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. ([#92938](https://github.com/kubernetes/kubernetes/pull/92938), [@lbernail](https://github.com/lbernail)) [SIG Network and Release] + - Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. ([#99471](https://github.com/kubernetes/kubernetes/pull/99471), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] + - Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99130](https://github.com/kubernetes/kubernetes/pull/99130), [@ayberk](https://github.com/ayberk)) [SIG Cloud Provider, Storage and Testing] + - Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99719](https://github.com/kubernetes/kubernetes/pull/99719), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider and Storage] + - Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. ([#98700](https://github.com/kubernetes/kubernetes/pull/98700), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider, Storage and Testing] + - OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. ([#98538](https://github.com/kubernetes/kubernetes/pull/98538), [@dims](https://github.com/dims)) [SIG Storage] + - Remove alpha `CSIMigrationXXComplete` flag and add alpha `InTreePluginXXUnregister` flag. Deprecate `CSIMigrationvSphereComplete` flag and it will be removed in v1.22. ([#98243](https://github.com/kubernetes/kubernetes/pull/98243), [@Jiawei0227](https://github.com/Jiawei0227)) + - Remove storage metrics `storage_operation_errors_total`, since we already have `storage_operation_status_count`.And add new field `status` for `storage_operation_duration_seconds`, so that we can know about all status storage operation latency. ([#98332](https://github.com/kubernetes/kubernetes/pull/98332), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] + - The metric `storage_operation_errors_total` is not removed, but is marked deprecated, and the metric `storage_operation_status_count` is marked deprecated. In both cases the `storage_operation_duration_seconds` metric can be used to recover equivalent counts (using `status=fail-unknown` in the case of `storage_operations_errors_total`). ([#99045](https://github.com/kubernetes/kubernetes/pull/99045), [@mattcary](https://github.com/mattcary)) + - `ServiceNodeExclusion`, `NodeDisruptionExclusion` and `LegacyNodeRoleBehavior` features have been promoted to GA. `ServiceNodeExclusion` and `NodeDisruptionExclusion` are now unconditionally enabled, while `LegacyNodeRoleBehavior` is unconditionally disabled. To prevent control plane nodes from being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. ([#97543](https://github.com/kubernetes/kubernetes/pull/97543), [@pacoxu](https://github.com/pacoxu)) + ## Changes by Kind ### Deprecation -- Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. ([#94624](https://github.com/kubernetes/kubernetes/pull/94624), [@dims](https://github.com/dims)) [SIG Node] -- Kubeadm: deprecate self-hosting support. The experimental command "kubeadm alpha self-hosting" is now deprecated and will be removed in a future release. ([#95125](https://github.com/kubernetes/kubernetes/pull/95125), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: graduate the "kubeadm alpha certs" command to a parent command "kubeadm certs". The command "kubeadm alpha certs" is deprecated and will be removed in a future release. Please migrate. ([#94938](https://github.com/kubernetes/kubernetes/pull/94938), [@yagonobre](https://github.com/yagonobre)) [SIG Cluster Lifecycle] -- Kubeadm: remove the deprecated "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. This change also removes the parent command "kubeadm alpha kubelet" as there are no more sub-commands under it for the time being. ([#94668](https://github.com/kubernetes/kubernetes/pull/94668), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: remove the deprecated --kubelet-config flag for the command "kubeadm upgrade node" ([#94869](https://github.com/kubernetes/kubernetes/pull/94869), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubectl: deprecate --delete-local-data ([#95076](https://github.com/kubernetes/kubernetes/pull/95076), [@dougsland](https://github.com/dougsland)) [SIG CLI, Cloud Provider and Scalability] -- Kubelet's deprecated endpoint `metrics/resource/v1alpha1` has been removed, please adopt `metrics/resource`. ([#94272](https://github.com/kubernetes/kubernetes/pull/94272), [@RainbowMango](https://github.com/RainbowMango)) [SIG Instrumentation and Node] -- Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration ([#94884](https://github.com/kubernetes/kubernetes/pull/94884), [@arghya88](https://github.com/arghya88)) [SIG Instrumentation and Scheduling] -- Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 ([#95001](https://github.com/kubernetes/kubernetes/pull/95001), [@arghya88](https://github.com/arghya88)) [SIG Instrumentation and Scheduling] -- Support 'controlplane' as a valid EgressSelection type in the EgressSelectorConfiguration API. 'Master' is deprecated and will be removed in v1.22. ([#95235](https://github.com/kubernetes/kubernetes/pull/95235), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] -- The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. ([#94090](https://github.com/kubernetes/kubernetes/pull/94090), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing] - +- Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. ([#98203](https://github.com/kubernetes/kubernetes/pull/98203), [@yuzhiquan](https://github.com/yuzhiquan)) +- Delete deprecated `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols` mixed procotol annotation in favor of the MixedProtocolLBService feature ([#97096](https://github.com/kubernetes/kubernetes/pull/97096), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Deprecate the `topologyKeys` field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. ([#96736](https://github.com/kubernetes/kubernetes/pull/96736), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] +- Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS ([#97336](https://github.com/kubernetes/kubernetes/pull/97336), [@maaoBit](https://github.com/maaoBit)) [SIG Network] +- Kubeadm: deprecated command "alpha selfhosting pivot" is now removed. ([#97627](https://github.com/kubernetes/kubernetes/pull/97627), [@knight42](https://github.com/knight42)) +- Kubeadm: graduate the command `kubeadm alpha kubeconfig user` to `kubeadm kubeconfig user`. The `kubeadm alpha kubeconfig user` command is deprecated now. ([#97583](https://github.com/kubernetes/kubernetes/pull/97583), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] +- Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. ([#97706](https://github.com/kubernetes/kubernetes/pull/97706), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] +- Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. ([#99646](https://github.com/kubernetes/kubernetes/pull/99646), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] +- Kubectl: The deprecated `kubectl alpha debug` command is removed. Use `kubectl debug` instead. ([#98111](https://github.com/kubernetes/kubernetes/pull/98111), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] +- Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97935](https://github.com/kubernetes/kubernetes/pull/97935), [@adeniyistephen](https://github.com/adeniyistephen)) [SIG Release and Testing] +- Remove deprecated `--generator, --replicas, --service-generator, --service-overrides, --schedule` from `kubectl run` + Deprecate `--serviceaccount, --hostport, --requests, --limits` in `kubectl run` ([#99732](https://github.com/kubernetes/kubernetes/pull/99732), [@soltysh](https://github.com/soltysh)) +- Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. ([#96447](https://github.com/kubernetes/kubernetes/pull/96447), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing] +- Removing experimental windows container hyper-v support with Docker ([#97141](https://github.com/kubernetes/kubernetes/pull/97141), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] +- Rename metrics `etcd_object_counts` to `apiserver_storage_object_counts` and mark it as stable. The original `etcd_object_counts` metrics name is marked as "Deprecated" and will be removed in the future. ([#99785](https://github.com/kubernetes/kubernetes/pull/99785), [@erain](https://github.com/erain)) [SIG API Machinery, Instrumentation and Testing] +- The GA TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled. Remove explicit use of those feature gates in CLI invocations. ([#97148](https://github.com/kubernetes/kubernetes/pull/97148), [@wawa0210](https://github.com/wawa0210)) [SIG Node] +- The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. ([#97171](https://github.com/kubernetes/kubernetes/pull/97171), [@deads2k](https://github.com/deads2k)) [SIG Auth and CLI] +- The `batch/v2alpha1` CronJob type definitions and clients are deprecated and removed. ([#96987](https://github.com/kubernetes/kubernetes/pull/96987), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] +- The `export` query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. ([#98312](https://github.com/kubernetes/kubernetes/pull/98312), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth and Testing] +- `audit.k8s.io/v1beta1` and `audit.k8s.io/v1alpha1` audit policy configuration and audit events are deprecated in favor of `audit.k8s.io/v1`, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with `--audit-policy-file`, or explicitly request alpha or beta audit events with `--audit-log-version` / `--audit-webhook-version` must update to use `audit.k8s.io/v1` and accept `audit.k8s.io/v1` events prior to v1.24. ([#98858](https://github.com/kubernetes/kubernetes/pull/98858), [@carlory](https://github.com/carlory)) [SIG Auth] +- `discovery.k8s.io/v1beta1` EndpointSlices are deprecated in favor of `discovery.k8s.io/v1`, and will no longer be served in Kubernetes v1.25. ([#100472](https://github.com/kubernetes/kubernetes/pull/100472), [@liggitt](https://github.com/liggitt)) +- `diskformat` storage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove `diskformat` parameter. vSphere CSI Driver does not support diskformat storageclass parameter. + + vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3. + + VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15. + + Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers. + + Support for these deprecations will be available till Kubernetes v1.24. ([#98546](https://github.com/kubernetes/kubernetes/pull/98546), [@divyenpatel](https://github.com/divyenpatel)) ### API Change -- `TokenRequest` and `TokenRequestProjection` features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The `TokenRequest` and `TokenRequestProjection` feature gates will be removed in v1.21. - - kubeadm's kube-apiserver Pod manifest now includes the following flags by default "--service-account-key-file", "--service-account-signing-key-file", "--service-account-issuer". ([#93258](https://github.com/kubernetes/kubernetes/pull/93258), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing] -- A new `nofuzz` go build tag now disables gofuzz support. Release binaries enable this. ([#92491](https://github.com/kubernetes/kubernetes/pull/92491), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery] -- Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest ([#95741](https://github.com/kubernetes/kubernetes/pull/95741), [@katiewasnothere](https://github.com/katiewasnothere)) [SIG Node] -- Add a `serving` and `terminating` condition to the EndpointSlice API. - `serving` tracks the readiness of endpoints regardless of their terminating state. This is distinct from `ready` since `ready` is only true when pods are not terminating. - `terminating` is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. ([#92968](https://github.com/kubernetes/kubernetes/pull/92968), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps and Network] -- Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. - It changes the dual-stack API wrt Service from a single ipFamily field to 3 - fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), - ipFamilies (a list of families assigned), and clusterIPs (inclusive of - clusterIP). Most users do not need to set anything at all, defaulting will - handle it for them. Services are single-stack unless the user asks for - dual-stack. This is all gated by the "IPv6DualStack" feature gate. ([#91824](https://github.com/kubernetes/kubernetes/pull/91824), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] -- Add support for hugepages to downward API ([#86102](https://github.com/kubernetes/kubernetes/pull/86102), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] -- Adds kubelet alpha feature, `GracefulNodeShutdown` which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. ([#96129](https://github.com/kubernetes/kubernetes/pull/96129), [@bobbypage](https://github.com/bobbypage)) [SIG Node] -- AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. ([#96327](https://github.com/kubernetes/kubernetes/pull/96327), [@robscott](https://github.com/robscott)) [SIG Apps and Network] -- Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter - Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. ([#92744](https://github.com/kubernetes/kubernetes/pull/92744), [@uablrek](https://github.com/uablrek)) [SIG Apps and Network] -- Certain fields on Service objects will be automatically cleared when changing the service's `type` to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. ([#95196](https://github.com/kubernetes/kubernetes/pull/95196), [@thockin](https://github.com/thockin)) [SIG API Machinery, Apps, Network and Testing] -- Document that ServiceTopology feature is required to use `service.spec.topologyKeys`. ([#96528](https://github.com/kubernetes/kubernetes/pull/96528), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] -- EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate. - - EndpointSlice topology field will be deprecated in an upcoming release. - - EndpointSlice "IP" address type is formally removed after being deprecated in Kubernetes 1.17. - - The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. ([#96440](https://github.com/kubernetes/kubernetes/pull/96440), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps and Network] -- External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ ([#92632](https://github.com/kubernetes/kubernetes/pull/92632), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Node and Testing] -- Fewer candidates are enumerated for preemption to improve performance in large clusters. ([#94814](https://github.com/kubernetes/kubernetes/pull/94814), [@adtac](https://github.com/adtac)) -- Fix conversions for custom metrics. ([#94481](https://github.com/kubernetes/kubernetes/pull/94481), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] -- GPU metrics provided by kubelet are now disabled by default. ([#95184](https://github.com/kubernetes/kubernetes/pull/95184), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) -- If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric `serviceaccount_stale_tokens_total` to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting `kube-apiserver` with flag `--service-account-extend-token-expiration=false` ([#96273](https://github.com/kubernetes/kubernetes/pull/96273), [@zshihang](https://github.com/zshihang)) [SIG API Machinery and Auth] -- Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. ([#94196](https://github.com/kubernetes/kubernetes/pull/94196), [@andrewsykim](https://github.com/andrewsykim)) [SIG Node and Release] -- Introduces a metric source for HPAs which allows scaling based on container resource usage. ([#90691](https://github.com/kubernetes/kubernetes/pull/90691), [@arjunrn](https://github.com/arjunrn)) [SIG API Machinery, Apps, Autoscaling and CLI] -- Kube-apiserver now deletes expired kube-apiserver Lease objects: - - The feature is under feature gate `APIServerIdentity`. - - A flag is added to kube-apiserver: `identity-lease-garbage-collection-check-period-seconds` ([#95895](https://github.com/kubernetes/kubernetes/pull/95895), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery, Apps, Auth and Testing] -- Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting `--volume-host-allow-local-loopback=false`, or from contacting specific CIDR ranges by setting `--volume-host-cidr-denylist` (for example, `--volume-host-cidr-denylist=127.0.0.1/28,feed::/16`) ([#91785](https://github.com/kubernetes/kubernetes/pull/91785), [@mattcary](https://github.com/mattcary)) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing] -- Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock ([#94603](https://github.com/kubernetes/kubernetes/pull/94603), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery, Apps, Cloud Provider and Scheduling] -- Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly ([#94182](https://github.com/kubernetes/kubernetes/pull/94182), [@mattfenwick](https://github.com/mattfenwick)) [SIG API Machinery, Apps, Auth, Network and Node] -- Move configurable fsgroup change policy for pods to beta ([#96376](https://github.com/kubernetes/kubernetes/pull/96376), [@gnufied](https://github.com/gnufied)) [SIG Apps and Storage] -- New flag is introduced, i.e. --topology-manager-scope=container|pod. - The default value is the "container" scope. ([#92967](https://github.com/kubernetes/kubernetes/pull/92967), [@cezaryzukowski](https://github.com/cezaryzukowski)) [SIG Instrumentation, Node and Testing] -- New parameter `defaultingType` for `PodTopologySpread` plugin allows to use k8s defined or user provided default constraints ([#95048](https://github.com/kubernetes/kubernetes/pull/95048), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] -- NodeAffinity plugin can be configured with AddedAffinity. ([#96202](https://github.com/kubernetes/kubernetes/pull/96202), [@alculquicondor](https://github.com/alculquicondor)) [SIG Node, Scheduling and Testing] -- Promote RuntimeClass feature to GA. - Promote node.k8s.io API groups from v1beta1 to v1. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] -- Reminder: The labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" are deprecated in favor of "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" respectively. All users of the "failure-domain.beta..." labels should switch to the "topology..." equivalents. ([#96033](https://github.com/kubernetes/kubernetes/pull/96033), [@thockin](https://github.com/thockin)) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing] -- Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. ([#93901](https://github.com/kubernetes/kubernetes/pull/93901), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing] -- Services will now have a `clusterIPs` field to go with `clusterIP`. `clusterIPs[0]` is a synonym for `clusterIP` and will be syncronized on create and update operations. ([#95894](https://github.com/kubernetes/kubernetes/pull/95894), [@thockin](https://github.com/thockin)) [SIG Network] -- The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. ([#91921](https://github.com/kubernetes/kubernetes/pull/91921), [@mtaufen](https://github.com/mtaufen)) [SIG Auth] -- The status of v1beta1 CRDs without "preserveUnknownFields:false" now shows a violation, "spec.preserveUnknownFields: Invalid value: true: must be false". ([#93078](https://github.com/kubernetes/kubernetes/pull/93078), [@vareti](https://github.com/vareti)) -- The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBService is enabled. The feature gate is disabled by default. The user has to enable it for the API Server. ([#94028](https://github.com/kubernetes/kubernetes/pull/94028), [@janosi](https://github.com/janosi)) [SIG API Machinery and Apps] -- This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in `CSIDriverSpec`. ([#93130](https://github.com/kubernetes/kubernetes/pull/93130), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing] -- Users can try the cronjob controller v2 using the feature gate. This will be the default controller in future releases. ([#93370](https://github.com/kubernetes/kubernetes/pull/93370), [@alaypatel07](https://github.com/alaypatel07)) [SIG API Machinery, Apps, Auth and Testing] -- VolumeSnapshotDataSource moves to GA in 1.20 release ([#95282](https://github.com/kubernetes/kubernetes/pull/95282), [@xing-yang](https://github.com/xing-yang)) [SIG Apps] -- WinOverlay feature graduated to beta ([#94807](https://github.com/kubernetes/kubernetes/pull/94807), [@ksubrmnn](https://github.com/ksubrmnn)) [SIG Windows] +- 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels. + 2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. ([#98582](https://github.com/kubernetes/kubernetes/pull/98582), [@ahg-g](https://github.com/ahg-g)) [SIG API Machinery, Apps, Auth and Testing] +- Add Probe-level terminationGracePeriodSeconds field ([#99375](https://github.com/kubernetes/kubernetes/pull/99375), [@ehashman](https://github.com/ehashman)) [SIG API Machinery, Apps, Node and Testing] +- Added `.spec.completionMode` field to Job, with accepted values `NonIndexed` (default) and `Indexed`. This is an alpha field and is only honored by servers with the `IndexedJob` feature gate enabled. ([#98441](https://github.com/kubernetes/kubernetes/pull/98441), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] +- Adds support for endPort field in NetworkPolicy ([#97058](https://github.com/kubernetes/kubernetes/pull/97058), [@rikatz](https://github.com/rikatz)) [SIG Apps and Network] +- CSIServiceAccountToken graduates to Beta and enabled by default. ([#99298](https://github.com/kubernetes/kubernetes/pull/99298), [@zshihang](https://github.com/zshihang)) +- Cluster admins can now turn off `/debug/pprof` and `/debug/flags/v` endpoint in kubelet by setting `enableProfilingHandler` and `enableDebugFlagsHandler` to `false` in the Kubelet configuration file. Options `enableProfilingHandler` and `enableDebugFlagsHandler` can be set to `true` only when `enableDebuggingHandlers` is also set to `true`. ([#98458](https://github.com/kubernetes/kubernetes/pull/98458), [@SaranBalaji90](https://github.com/SaranBalaji90)) +- DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. ([#96441](https://github.com/kubernetes/kubernetes/pull/96441), [@smarterclayton](https://github.com/smarterclayton)) [SIG Apps and Testing] +- Enable SPDY pings to keep connections alive, so that `kubectl exec` and `kubectl portforward` won't be interrupted. ([#97083](https://github.com/kubernetes/kubernetes/pull/97083), [@knight42](https://github.com/knight42)) [SIG API Machinery and CLI] +- FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). ([#99661](https://github.com/kubernetes/kubernetes/pull/99661), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery, Auth and Testing] +- Fixes server-side apply for APIService resources. ([#98576](https://github.com/kubernetes/kubernetes/pull/98576), [@kevindelgado](https://github.com/kevindelgado)) +- Generic ephemeral volumes are beta. ([#99643](https://github.com/kubernetes/kubernetes/pull/99643), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing] +- Hugepages request values are limited to integer multiples of the page size. ([#98515](https://github.com/kubernetes/kubernetes/pull/98515), [@lala123912](https://github.com/lala123912)) [SIG Apps] +- Implement the GetAvailableResources in the podresources API. ([#95734](https://github.com/kubernetes/kubernetes/pull/95734), [@fromanirh](https://github.com/fromanirh)) [SIG Instrumentation, Node and Testing] +- IngressClass resource can now reference a resource in a specific namespace + for implementation-specific configuration (previously only Cluster-level resources were allowed). + This feature can be enabled using the IngressClassNamespacedParams feature gate. ([#99275](https://github.com/kubernetes/kubernetes/pull/99275), [@hbagdi](https://github.com/hbagdi)) +- Jobs API has a new `.spec.suspend` field that can be used to suspend and resume Jobs. This is an alpha field which is only honored by servers with the `SuspendJob` feature gate enabled. ([#98727](https://github.com/kubernetes/kubernetes/pull/98727), [@adtac](https://github.com/adtac)) +- Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default. ([#99735](https://github.com/kubernetes/kubernetes/pull/99735), [@bobbypage](https://github.com/bobbypage)) +- Kubernetes is now built using go1.15.7 ([#98363](https://github.com/kubernetes/kubernetes/pull/98363), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Node, Release and Testing] +- Namespace API objects now have a `kubernetes.io/metadata.name` label matching their metadata.name field to allow selecting any namespace by its name using a label selector. ([#96968](https://github.com/kubernetes/kubernetes/pull/96968), [@jayunit100](https://github.com/jayunit100)) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing] +- One new field "InternalTrafficPolicy" in Service is added. + It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. + "Cluster" routes internal traffic to a Service to all endpoints. + "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. + The default value is "Cluster". ([#96600](https://github.com/kubernetes/kubernetes/pull/96600), [@maplain](https://github.com/maplain)) [SIG API Machinery, Apps and Network] +- PodDisruptionBudget API objects can now contain conditions in status. ([#98127](https://github.com/kubernetes/kubernetes/pull/98127), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation] +- PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled ([#98918](https://github.com/kubernetes/kubernetes/pull/98918), [@pohly](https://github.com/pohly)) [SIG Auth and Security] +- Promote CronJobs to batch/v1 ([#99423](https://github.com/kubernetes/kubernetes/pull/99423), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] +- Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set `immutable` field in Secret or ConfigMap object to mark their contents as immutable. ([#97615](https://github.com/kubernetes/kubernetes/pull/97615), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps, Architecture, Node and Testing] +- Remove support for building Kubernetes with bazel. ([#99561](https://github.com/kubernetes/kubernetes/pull/99561), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows] +- Scheduler extender filter interface now can report unresolvable failed nodes in the new field `FailedAndUnresolvableNodes` of `ExtenderFilterResult` struct. Nodes in this map will be skipped in the preemption phase. ([#92866](https://github.com/kubernetes/kubernetes/pull/92866), [@cofyc](https://github.com/cofyc)) [SIG Scheduling] +- Services can specify loadBalancerClass to use a custom load balancer ([#98277](https://github.com/kubernetes/kubernetes/pull/98277), [@XudongLiuHarold](https://github.com/XudongLiuHarold)) +- Storage capacity tracking (= the CSIStorageCapacity feature) graduates to Beta and enabled by default, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated ([#99641](https://github.com/kubernetes/kubernetes/pull/99641), [@pohly](https://github.com/pohly)) +- Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. ([#98812](https://github.com/kubernetes/kubernetes/pull/98812), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] +- The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default. + - This changes the tokens provided to containers at `/var/run/secrets/kubernetes.io/serviceaccount/token` to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted. + - Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. `k8s.io/client-go` version v11.0.0+ and v0.15.0+ reload tokens automatically. + - By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric `serviceaccount_stale_tokens_total` can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting `kube-apiserver` with `--service-account-extend-token-expiration=false`. ([#95667](https://github.com/kubernetes/kubernetes/pull/95667), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle and Testing] +- The EndpointSlice Controllers are now GA. The `EndpointSliceController` will not populate the `deprecatedTopology` field and will only provide topology information through the `zone` and `nodeName` fields. ([#99870](https://github.com/kubernetes/kubernetes/pull/99870), [@swetharepakula](https://github.com/swetharepakula)) +- The Endpoints controller will now set the `endpoints.kubernetes.io/over-capacity` annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. ([#99975](https://github.com/kubernetes/kubernetes/pull/99975), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector (`{}`) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. ([#99290](https://github.com/kubernetes/kubernetes/pull/99290), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing] +- The `EndpointSlice` API is now GA. The `EndpointSlice` topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The `discovery.k8s.io/v1alpha1` API is removed. ([#99662](https://github.com/kubernetes/kubernetes/pull/99662), [@swetharepakula](https://github.com/swetharepakula)) +- The `controller.kubernetes.io/pod-deletion-cost` annotation can be set to offer a hint on the cost of deleting a `Pod` compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. ([#99163](https://github.com/kubernetes/kubernetes/pull/99163), [@ahg-g](https://github.com/ahg-g)) +- The kube-apiserver now resets `managedFields` that got corrupted by a mutating admission controller. ([#98074](https://github.com/kubernetes/kubernetes/pull/98074), [@kwiesmueller](https://github.com/kwiesmueller)) +- Topology Aware Hints are now available in alpha and can be enabled with the `TopologyAwareHints` feature gate. ([#99522](https://github.com/kubernetes/kubernetes/pull/99522), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing] +- Users might specify the `kubectl.kubernetes.io/default-exec-container` annotation in a Pod to preselect container for kubectl commands. ([#97099](https://github.com/kubernetes/kubernetes/pull/97099), [@pacoxu](https://github.com/pacoxu)) [SIG CLI] ### Feature -- **Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.**: -- A new metric `apiserver_request_filter_duration_seconds` has been introduced that - measures request filter latency in seconds. ([#95207](https://github.com/kubernetes/kubernetes/pull/95207), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Instrumentation] -- A new set of alpha metrics are reported by the Kubernetes scheduler under the `/metrics/resources` endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. ([#94866](https://github.com/kubernetes/kubernetes/pull/94866), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Instrumentation, Node and Scheduling] -- Add --experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs ([#96370](https://github.com/kubernetes/kubernetes/pull/96370), [@serathius](https://github.com/serathius)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] -- Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. - This feature allows the storage migrator to manage storage migration for built-in resources. - Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. ([#93873](https://github.com/kubernetes/kubernetes/pull/93873), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery, Auth and Testing] -- Add a metric for time taken to perform recursive permission change ([#95866](https://github.com/kubernetes/kubernetes/pull/95866), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] -- Add a new `vSphere` metric: `cloudprovider_vsphere_vcenter_versions`. It's content show `vCenter` hostnames with the associated server version. ([#94526](https://github.com/kubernetes/kubernetes/pull/94526), [@Danil-Grigorev](https://github.com/Danil-Grigorev)) [SIG Cloud Provider and Instrumentation] -- Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node there by disrupting kubelet process. ([#96051](https://github.com/kubernetes/kubernetes/pull/96051), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Node and Windows] -- Add feature to size memory backed volumes ([#94444](https://github.com/kubernetes/kubernetes/pull/94444), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Storage and Testing] -- Add foreground cascading deletion to kubectl with the new `kubectl delete foreground|background|orphan` option. ([#93384](https://github.com/kubernetes/kubernetes/pull/93384), [@zhouya0](https://github.com/zhouya0)) -- Add metrics for azure service operations (route and loadbalancer). ([#94124](https://github.com/kubernetes/kubernetes/pull/94124), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider and Instrumentation] -- Add network rule support in Azure account creation. ([#94239](https://github.com/kubernetes/kubernetes/pull/94239), [@andyzhangx](https://github.com/andyzhangx)) -- Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. ([#92466](https://github.com/kubernetes/kubernetes/pull/92466), [@mborsz](https://github.com/mborsz)) [SIG API Machinery, Auth and Instrumentation] -- Add pod_ based CPU and memory metrics to Kubelet's /metrics/resource endpoint ([#95839](https://github.com/kubernetes/kubernetes/pull/95839), [@egernst](https://github.com/egernst)) [SIG Instrumentation, Node and Testing] -- Added `get-users` and `delete-user` to the `kubectl config` subcommand ([#89840](https://github.com/kubernetes/kubernetes/pull/89840), [@eddiezane](https://github.com/eddiezane)) [SIG CLI] -- Added counter metric "apiserver_request_self" to count API server self-requests with labels for verb, resource, and subresource. ([#94288](https://github.com/kubernetes/kubernetes/pull/94288), [@LogicalShark](https://github.com/LogicalShark)) [SIG API Machinery, Auth, Instrumentation and Scheduling] -- Added new k8s.io/component-helpers repository providing shared helper code for (core) components. ([#92507](https://github.com/kubernetes/kubernetes/pull/92507), [@ingvagabund](https://github.com/ingvagabund)) [SIG Apps, Node, Release and Scheduling] -- Adds `create ingress` command to `kubectl` ([#78153](https://github.com/kubernetes/kubernetes/pull/78153), [@amimof](https://github.com/amimof)) [SIG CLI and Network] -- Adds a headless service on node-local-cache addon. ([#88412](https://github.com/kubernetes/kubernetes/pull/88412), [@stafot](https://github.com/stafot)) [SIG Cloud Provider and Network] -- Allow cross compilation of kubernetes on different platforms. ([#94403](https://github.com/kubernetes/kubernetes/pull/94403), [@bnrjee](https://github.com/bnrjee)) [SIG Release] -- Azure: Support multiple services sharing one IP address ([#94991](https://github.com/kubernetes/kubernetes/pull/94991), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in list will continue being preserved, and fail validation if not nullable. ([#95423](https://github.com/kubernetes/kubernetes/pull/95423), [@apelisse](https://github.com/apelisse)) [SIG API Machinery] -- Changed: default "Accept: */*" header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) ([#95641](https://github.com/kubernetes/kubernetes/pull/95641), [@fonsecas72](https://github.com/fonsecas72)) [SIG Network and Node] -- Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. ([#95489](https://github.com/kubernetes/kubernetes/pull/95489), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery and Auth] -- Command to start network proxy changes from 'KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh' to 'KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh' ([#92669](https://github.com/kubernetes/kubernetes/pull/92669), [@Jefftree](https://github.com/Jefftree)) [SIG Cloud Provider] -- Configure AWS LoadBalancer health check protocol via service annotations. ([#94546](https://github.com/kubernetes/kubernetes/pull/94546), [@kishorj](https://github.com/kishorj)) -- DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. ([#95631](https://github.com/kubernetes/kubernetes/pull/95631), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling and Testing] -- E2e test for PodFsGroupChangePolicy ([#96247](https://github.com/kubernetes/kubernetes/pull/96247), [@saikat-royc](https://github.com/saikat-royc)) [SIG Storage and Testing] -- Ephemeral containers now apply the same API defaults as initContainers and containers ([#94896](https://github.com/kubernetes/kubernetes/pull/94896), [@wawa0210](https://github.com/wawa0210)) [SIG Apps and CLI] -- Gradudate the Pod Resources API to G.A - Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API ([#92165](https://github.com/kubernetes/kubernetes/pull/92165), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Instrumentation, Node and Testing] -- In dual-stack bare-metal clusters, you can now pass dual-stack IPs to `kubelet --node-ip`. - eg: `kubelet --node-ip 10.1.0.5,fd01::0005`. This is not yet supported for non-bare-metal - clusters. - - In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods - will now get dual-stack PodIPs. ([#95239](https://github.com/kubernetes/kubernetes/pull/95239), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] -- Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. ([#95603](https://github.com/kubernetes/kubernetes/pull/95603), [@soltysh](https://github.com/soltysh)) [SIG API Machinery] -- Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. ([#91977](https://github.com/kubernetes/kubernetes/pull/91977), [@varunmar](https://github.com/varunmar)) [SIG Cloud Provider] -- Kube-apiserver now maintains a Lease object to identify itself: - - The feature is under feature gate `APIServerIdentity`. - - Two flags are added to kube-apiserver: `identity-lease-duration-seconds`, `identity-lease-renew-interval-seconds` ([#95533](https://github.com/kubernetes/kubernetes/pull/95533), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] -- Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with `--etcd-healthcheck-timeout`. The default timeout is 2 seconds, matching the previous behavior. ([#93244](https://github.com/kubernetes/kubernetes/pull/93244), [@Sh4d1](https://github.com/Sh4d1)) [SIG API Machinery] -- Kube-apiserver: added support for compressing rotated audit log files with `--audit-log-compress` ([#94066](https://github.com/kubernetes/kubernetes/pull/94066), [@lojies](https://github.com/lojies)) [SIG API Machinery and Auth] -- Kubeadm now prints warnings instead of throwing errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate. ([#94504](https://github.com/kubernetes/kubernetes/pull/94504), [@neolit123](https://github.com/neolit123)) -- Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM ([#93275](https://github.com/kubernetes/kubernetes/pull/93275), [@xlgao-zju](https://github.com/xlgao-zju)) [SIG Cluster Lifecycle] -- Kubeadm: add the "--cluster-name" flag to the "kubeadm alpha kubeconfig user" to allow configuring the cluster name in the generated kubeconfig file ([#93992](https://github.com/kubernetes/kubernetes/pull/93992), [@prabhu43](https://github.com/prabhu43)) [SIG Cluster Lifecycle] -- Kubeadm: add the "--kubeconfig" flag to the "kubeadm init phase upload-certs" command to allow users to pass a custom location for a kubeconfig file. ([#94765](https://github.com/kubernetes/kubernetes/pull/94765), [@zhanw15](https://github.com/zhanw15)) [SIG Cluster Lifecycle] -- Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default ([#94479](https://github.com/kubernetes/kubernetes/pull/94479), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] -- Kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags: - - apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration. - - cluster-name: use clusterName from ClusterConfiguration - - cert-dir: use certificatesDir from ClusterConfiguration ([#94879](https://github.com/kubernetes/kubernetes/pull/94879), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] -- Kubectl create now supports creating ingress objects. ([#94327](https://github.com/kubernetes/kubernetes/pull/94327), [@rikatz](https://github.com/rikatz)) [SIG CLI and Network] -- Kubectl rollout history sts/sts-name --revision=some-revision will start showing the detailed view of the sts on that specified revision ([#86506](https://github.com/kubernetes/kubernetes/pull/86506), [@dineshba](https://github.com/dineshba)) [SIG CLI] -- Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. ([#95292](https://github.com/kubernetes/kubernetes/pull/95292), [@dougsland](https://github.com/dougsland)) [SIG CLI] -- Kubemark now supports both real and hollow nodes in a single cluster. ([#93201](https://github.com/kubernetes/kubernetes/pull/93201), [@ellistarn](https://github.com/ellistarn)) [SIG Scalability] -- Kubernetes E2E test image manifest lists now contain Windows images. ([#77398](https://github.com/kubernetes/kubernetes/pull/77398), [@claudiubelu](https://github.com/claudiubelu)) [SIG Testing and Windows] -- Kubernetes is now built using go1.15.2 - - build: Update to k/repo-infra@v0.1.1 (supports go1.15.2) - - build: Use go-runner:buster-v2.0.1 (built using go1.15.1) - - bazel: Replace --features with Starlark build settings flag - - hack/lib/util.sh: some bash cleanups - - - switched one spot to use kube::logging - - make kube::util::find-binary return an error when it doesn't find - anything so that hack scripts fail fast instead of with '' binary not - found errors. - - this required deleting some genfeddoc stuff. the binary no longer - exists in k/k repo since we removed federation/, and I don't see it - in https://github.com/kubernetes-sigs/kubefed/ either. I'm assuming - that it's gone for good now. - - - bazel: output go_binary rule directly from go_binary_conditional_pure - - From: [@mikedanese](https://github.com/mikedanese): - Instead of aliasing. Aliases are annoying in a number of ways. This is - specifically bugging me now because they make the action graph harder to - analyze programmatically. By using aliases here, we would need to handle - potentially aliased go_binary targets and dereference to the effective - target. - - The comment references an issue with `pure = select(...)` which appears - to be resolved considering this now builds. - - - make kube::util::find-binary not dependent on bazel-out/ structure - - Implement an aspect that outputs go_build_mode metadata for go binaries, - and use that during binary selection. ([#94449](https://github.com/kubernetes/kubernetes/pull/94449), [@justaugustus](https://github.com/justaugustus)) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing] -- Kubernetes is now built using go1.15.5 - - build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) ([#95776](https://github.com/kubernetes/kubernetes/pull/95776), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] -- New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used ([#95539](https://github.com/kubernetes/kubernetes/pull/95539), [@soulxu](https://github.com/soulxu)) [SIG Scheduling] -- Only update Azure data disks when attach/detach ([#94265](https://github.com/kubernetes/kubernetes/pull/94265), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Promote SupportNodePidsLimit to GA to provide node-to-pod PID isolation. - Promote SupportPodPidsLimit to GA to provide ability to limit PIDs per pod. ([#94140](https://github.com/kubernetes/kubernetes/pull/94140), [@derekwaynecarr](https://github.com/derekwaynecarr)) -- SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. - Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, - and note that some cloud platforms and network plugins do not support SCTP traffic. ([#95566](https://github.com/kubernetes/kubernetes/pull/95566), [@danwinship](https://github.com/danwinship)) [SIG Apps and Network] -- Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. ([#96071](https://github.com/kubernetes/kubernetes/pull/96071), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] -- Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. ([#93534](https://github.com/kubernetes/kubernetes/pull/93534), [@everpeace](https://github.com/everpeace)) [SIG Scheduling and Testing] -- SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled ([#95448](https://github.com/kubernetes/kubernetes/pull/95448), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] -- Send GCE node startup scripts logs to console and journal. ([#95311](https://github.com/kubernetes/kubernetes/pull/95311), [@karan](https://github.com/karan)) -- SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. ([#95267](https://github.com/kubernetes/kubernetes/pull/95267), [@javidiaz](https://github.com/javidiaz)) [SIG Node] -- Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] ([#94114](https://github.com/kubernetes/kubernetes/pull/94114), [@MarcPow](https://github.com/MarcPow)) [SIG Cloud Provider] -- Support custom tags for cloud provider managed resources ([#96450](https://github.com/kubernetes/kubernetes/pull/96450), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Support customize load balancer health probe protocol and request path ([#96338](https://github.com/kubernetes/kubernetes/pull/96338), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added the pause:3.4 image. ([#91452](https://github.com/kubernetes/kubernetes/pull/91452), [@claudiubelu](https://github.com/claudiubelu)) [SIG Node, Release and Windows] -- Support multiple standard load balancers in one cluster ([#96111](https://github.com/kubernetes/kubernetes/pull/96111), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- The beta `RootCAConfigMap` feature gate is enabled by default and causes kube-controller-manager to publish a "kube-root-ca.crt" ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. ([#96197](https://github.com/kubernetes/kubernetes/pull/96197), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Apps, Auth and Testing] -- The kubelet_runtime_operations_duration_seconds metric buckets were set to 0.005 0.0125 0.03125 0.078125 0.1953125 0.48828125 1.220703125 3.0517578125 7.62939453125 19.073486328125 47.6837158203125 119.20928955078125 298.0232238769531 and 745.0580596923828 seconds ([#96054](https://github.com/kubernetes/kubernetes/pull/96054), [@alvaroaleman](https://github.com/alvaroaleman)) [SIG Instrumentation and Node] -- There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. ([#95719](https://github.com/kubernetes/kubernetes/pull/95719), [@tsmetana](https://github.com/tsmetana)) [SIG Apps, Instrumentation, Storage and Testing] -- Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer ([#95863](https://github.com/kubernetes/kubernetes/pull/95863), [@RaunakShah](https://github.com/RaunakShah)) [SIG Cloud Provider, Storage and Testing] -- Warns user when executing kubectl apply/diff to resource currently being deleted. ([#95544](https://github.com/kubernetes/kubernetes/pull/95544), [@SaiHarshaK](https://github.com/SaiHarshaK)) [SIG CLI] -- `kubectl alpha debug` has graduated to beta and is now `kubectl debug`. ([#96138](https://github.com/kubernetes/kubernetes/pull/96138), [@verb](https://github.com/verb)) [SIG CLI and Testing] -- `kubectl debug` gains support for changing container images when copying a pod for debugging, similar to how `kubectl set image` works. See `kubectl help debug` for more information. ([#96058](https://github.com/kubernetes/kubernetes/pull/96058), [@verb](https://github.com/verb)) [SIG CLI] - -### Documentation - -- Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList ([#95117](https://github.com/kubernetes/kubernetes/pull/95117), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] -- Kubelet: remove alpha warnings for CNI flags. ([#94508](https://github.com/kubernetes/kubernetes/pull/94508), [@andrewsykim](https://github.com/andrewsykim)) [SIG Network and Node] -- Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers: - - removes experimental warning for InstancesV2 - - document that implementation of InstancesV2 will disable calls to Zones - - deprecate Zones in favor of InstancesV2 ([#96397](https://github.com/kubernetes/kubernetes/pull/96397), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] - -### Failing Test - -- Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources ([#96742](https://github.com/kubernetes/kubernetes/pull/96742), [@spencerhance](https://github.com/spencerhance)) [SIG Network and Testing] -- The Conformance test "validates that there is no conflict between pods with same hostPort but different hostIP and protocol" now validates the connectivity to each hostPort, in addition to the functionality. ([#96627](https://github.com/kubernetes/kubernetes/pull/96627), [@aojea](https://github.com/aojea)) [SIG Scheduling and Testing] - -### Bug or Regression - -- Add kubectl wait --ignore-not-found flag ([#90969](https://github.com/kubernetes/kubernetes/pull/90969), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] -- Added support to kube-proxy for externalTrafficPolicy=Local setting via Direct Server Return (DSR) load balancers on Windows. ([#93166](https://github.com/kubernetes/kubernetes/pull/93166), [@elweb9858](https://github.com/elweb9858)) [SIG Network] -- Alter wording to describe pods using a pvc ([#95635](https://github.com/kubernetes/kubernetes/pull/95635), [@RaunakShah](https://github.com/RaunakShah)) [SIG CLI] -- An issues preventing volume expand controller to annotate the PVC with `volume.kubernetes.io/storage-resizer` when the PVC StorageClass is already updated to the out-of-tree provisioner is now fixed. ([#94489](https://github.com/kubernetes/kubernetes/pull/94489), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG API Machinery, Apps and Storage] -- Azure ARM client: don't segfault on empty response and http error ([#94078](https://github.com/kubernetes/kubernetes/pull/94078), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] -- Azure armclient backoff step defaults to 1 (no retry). ([#94180](https://github.com/kubernetes/kubernetes/pull/94180), [@feiskyer](https://github.com/feiskyer)) -- Azure: fix a bug that kube-controller-manager would panic if wrong Azure VMSS name is configured ([#94306](https://github.com/kubernetes/kubernetes/pull/94306), [@knight42](https://github.com/knight42)) [SIG Cloud Provider] -- Both apiserver_request_duration_seconds metrics and RequestReceivedTimestamp fields of an audit event now take into account the time a request spends in the apiserver request filters. ([#94903](https://github.com/kubernetes/kubernetes/pull/94903), [@tkashem](https://github.com/tkashem)) -- Build/lib/release: Explicitly use '--platform' in building server images - - When we switched to go-runner for building the apiserver, - controller-manager, and scheduler server components, we no longer - reference the individual architectures in the image names, specifically - in the 'FROM' directive of the server image Dockerfiles. - - As a result, server images for non-amd64 images copy in the go-runner - amd64 binary instead of the go-runner that matches that architecture. - - This commit explicitly sets the '--platform=linux/${arch}' to ensure - we're pulling the correct go-runner arch from the manifest list. - - Before: - `FROM ${base_image}` - - After: - `FROM --platform=linux/${arch} ${base_image}` ([#94552](https://github.com/kubernetes/kubernetes/pull/94552), [@justaugustus](https://github.com/justaugustus)) [SIG Release] -- Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ ([#96716](https://github.com/kubernetes/kubernetes/pull/96716), [@tosi3k](https://github.com/tosi3k)) [SIG Cloud Provider, Scalability and Testing] -- CSIDriver object can be deployed during volume attachment. ([#93710](https://github.com/kubernetes/kubernetes/pull/93710), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Node, Storage and Testing] -- Ceph RBD volume expansion now works even when ceph.conf was not provided. ([#92027](https://github.com/kubernetes/kubernetes/pull/92027), [@juliantaylor](https://github.com/juliantaylor)) -- Change plugin name in fsgroupapplymetrics of csi and flexvolume to distinguish different driver ([#95892](https://github.com/kubernetes/kubernetes/pull/95892), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation, Storage and Testing] -- Change the calculation of pod UIDs so that static pods get a unique value - will cause all containers to be killed and recreated after in-place upgrade. ([#87461](https://github.com/kubernetes/kubernetes/pull/87461), [@bboreham](https://github.com/bboreham)) [SIG Node] -- Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. ([#94916](https://github.com/kubernetes/kubernetes/pull/94916), [@smileusd](https://github.com/smileusd)) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing] -- Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. ([#96727](https://github.com/kubernetes/kubernetes/pull/96727), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] -- Clear UDP conntrack entry on endpoint changes when using nodeport ([#71573](https://github.com/kubernetes/kubernetes/pull/71573), [@JacobTanenbaum](https://github.com/JacobTanenbaum)) [SIG Network] -- Cloud node controller: handle empty providerID from getProviderID ([#95342](https://github.com/kubernetes/kubernetes/pull/95342), [@nicolehanjing](https://github.com/nicolehanjing)) [SIG Cloud Provider] -- Disable watchcache for events ([#96052](https://github.com/kubernetes/kubernetes/pull/96052), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] -- Disabled `LocalStorageCapacityIsolation` feature gate is honored during scheduling. ([#96092](https://github.com/kubernetes/kubernetes/pull/96092), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] -- Do not fail sorting empty elements. ([#94666](https://github.com/kubernetes/kubernetes/pull/94666), [@soltysh](https://github.com/soltysh)) [SIG CLI] -- Dual-stack: make nodeipam compatible with existing single-stack clusters when dual-stack feature gate become enabled by default ([#90439](https://github.com/kubernetes/kubernetes/pull/90439), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery] -- Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. ([#96185](https://github.com/kubernetes/kubernetes/pull/96185), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery and Testing] -- Endpoint slice controller now mirrors parent's service label to its corresponding endpoint slices. ([#94443](https://github.com/kubernetes/kubernetes/pull/94443), [@aojea](https://github.com/aojea)) -- Ensure getPrimaryInterfaceID not panic when network interfaces for Azure VMSS are null ([#94355](https://github.com/kubernetes/kubernetes/pull/94355), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] -- Exposes and sets a default timeout for the SubjectAccessReview client for DelegatingAuthorizationOptions ([#95725](https://github.com/kubernetes/kubernetes/pull/95725), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Cloud Provider] -- Exposes and sets a default timeout for the TokenReview client for DelegatingAuthenticationOptions ([#96217](https://github.com/kubernetes/kubernetes/pull/96217), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Cloud Provider] -- Fix CVE-2020-8555 for Quobyte client connections. ([#95206](https://github.com/kubernetes/kubernetes/pull/95206), [@misterikkit](https://github.com/misterikkit)) [SIG Storage] -- Fix IP fragmentation of UDP and TCP packets not supported issues on LoadBalancer rules ([#96464](https://github.com/kubernetes/kubernetes/pull/96464), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Fix a bug that DefaultPreemption plugin is disabled when using (legacy) scheduler policy. ([#96439](https://github.com/kubernetes/kubernetes/pull/96439), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] -- Fix a bug where loadbalancer deletion gets stuck because of missing resource group. ([#93962](https://github.com/kubernetes/kubernetes/pull/93962), [@phiphi282](https://github.com/phiphi282)) -- Fix a concurrent map writes error in kubelet ([#93773](https://github.com/kubernetes/kubernetes/pull/93773), [@knight42](https://github.com/knight42)) [SIG Node] -- Fix a panic in `kubectl debug` when a pod has multiple init or ephemeral containers. ([#94580](https://github.com/kubernetes/kubernetes/pull/94580), [@kiyoshim55](https://github.com/kiyoshim55)) -- Fix a regression where kubeadm bails out with a fatal error when an optional version command line argument is supplied to the "kubeadm upgrade plan" command ([#94421](https://github.com/kubernetes/kubernetes/pull/94421), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] -- Fix azure disk attach failure for disk size bigger than 4TB ([#95463](https://github.com/kubernetes/kubernetes/pull/95463), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix azure disk data loss issue on Windows when unmount disk ([#95456](https://github.com/kubernetes/kubernetes/pull/95456), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Fix azure file migration panic ([#94853](https://github.com/kubernetes/kubernetes/pull/94853), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix bug in JSON path parser where an error occurs when a range is empty ([#95933](https://github.com/kubernetes/kubernetes/pull/95933), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] -- Fix client-go prometheus metrics to correctly present the API path accessed in some environments. ([#74363](https://github.com/kubernetes/kubernetes/pull/74363), [@aanm](https://github.com/aanm)) [SIG API Machinery] -- Fix detach azure disk issue when vm not exist ([#95177](https://github.com/kubernetes/kubernetes/pull/95177), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix etcd_object_counts metric reported by kube-apiserver ([#94773](https://github.com/kubernetes/kubernetes/pull/94773), [@tkashem](https://github.com/tkashem)) [SIG API Machinery] -- Fix incorrectly reported verbs for kube-apiserver metrics for CRD objects ([#93523](https://github.com/kubernetes/kubernetes/pull/93523), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] -- Fix k8s.io/apimachinery/pkg/api/meta.SetStatusCondition to update ObservedGeneration ([#95961](https://github.com/kubernetes/kubernetes/pull/95961), [@KnicKnic](https://github.com/KnicKnic)) [SIG API Machinery] -- Fix kubectl SchemaError on CRDs with schema using x-kubernetes-preserve-unknown-fields on array types. ([#94888](https://github.com/kubernetes/kubernetes/pull/94888), [@sttts](https://github.com/sttts)) [SIG API Machinery] -- Fix memory leak in kube-apiserver when underlying time goes forth and back. ([#96266](https://github.com/kubernetes/kubernetes/pull/96266), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery] -- Fix missing csi annotations on node during parallel csinode update. ([#94389](https://github.com/kubernetes/kubernetes/pull/94389), [@pacoxu](https://github.com/pacoxu)) [SIG Storage] -- Fix network_programming_latency metric reporting for Endpoints/EndpointSlice deletions, where we don't have correct timestamp ([#95363](https://github.com/kubernetes/kubernetes/pull/95363), [@wojtek-t](https://github.com/wojtek-t)) [SIG Network and Scalability] -- Fix paging issues when Azure API returns empty values with non-empty nextLink ([#96211](https://github.com/kubernetes/kubernetes/pull/96211), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] -- Fix pull image error from multiple ACRs using azure managed identity ([#96355](https://github.com/kubernetes/kubernetes/pull/96355), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix race condition on timeCache locks. ([#94751](https://github.com/kubernetes/kubernetes/pull/94751), [@auxten](https://github.com/auxten)) -- Fix regression on `kubectl portforward` when TCP and UCP services were configured on the same port. ([#94728](https://github.com/kubernetes/kubernetes/pull/94728), [@amorenoz](https://github.com/amorenoz)) -- Fix scheduler cache snapshot when a Node is deleted before its Pods ([#95130](https://github.com/kubernetes/kubernetes/pull/95130), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] -- Fix the `cloudprovider_azure_api_request_duration_seconds` metric buckets to correctly capture the latency metrics. Previously, the majority of the calls would fall in the "+Inf" bucket. ([#94873](https://github.com/kubernetes/kubernetes/pull/94873), [@marwanad](https://github.com/marwanad)) [SIG Cloud Provider and Instrumentation] -- Fix vSphere volumes that could be erroneously attached to wrong node ([#96224](https://github.com/kubernetes/kubernetes/pull/96224), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] -- Fix verb & scope reporting for kube-apiserver metrics (LIST reported instead of GET) ([#95562](https://github.com/kubernetes/kubernetes/pull/95562), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] -- Fix vsphere detach failure for static PVs ([#95447](https://github.com/kubernetes/kubernetes/pull/95447), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] -- Fix: azure disk resize error if source does not exist ([#93011](https://github.com/kubernetes/kubernetes/pull/93011), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix: detach azure disk broken on Azure Stack ([#94885](https://github.com/kubernetes/kubernetes/pull/94885), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix: resize Azure disk issue when it's in attached state ([#96705](https://github.com/kubernetes/kubernetes/pull/96705), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix: smb valid path error ([#95583](https://github.com/kubernetes/kubernetes/pull/95583), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] -- Fix: use sensitiveOptions on Windows mount ([#94126](https://github.com/kubernetes/kubernetes/pull/94126), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Fixed a bug causing incorrect formatting of `kubectl describe ingress`. ([#94985](https://github.com/kubernetes/kubernetes/pull/94985), [@howardjohn](https://github.com/howardjohn)) [SIG CLI and Network] -- Fixed a bug in client-go where new clients with customized `Dial`, `Proxy`, `GetCert` config may get stale HTTP transports. ([#95427](https://github.com/kubernetes/kubernetes/pull/95427), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] -- Fixed a bug that prevents kubectl to validate CRDs with schema using x-kubernetes-preserve-unknown-fields on object fields. ([#96369](https://github.com/kubernetes/kubernetes/pull/96369), [@gautierdelorme](https://github.com/gautierdelorme)) [SIG API Machinery and Testing] -- Fixed a bug that prevents the use of ephemeral containers in the presence of a validating admission webhook. ([#94685](https://github.com/kubernetes/kubernetes/pull/94685), [@verb](https://github.com/verb)) [SIG Node and Testing] -- Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. ([#96421](https://github.com/kubernetes/kubernetes/pull/96421), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG API Machinery and Instrumentation] -- Fixed a bug where improper storage and comparison of endpoints led to excessive API traffic from the endpoints controller ([#94112](https://github.com/kubernetes/kubernetes/pull/94112), [@damemi](https://github.com/damemi)) [SIG Apps, Network and Testing] -- Fixed a regression which prevented pods with `docker/default` seccomp annotations from being created in 1.19 if a PodSecurityPolicy was in place which did not allow `runtime/default` seccomp profiles. ([#95985](https://github.com/kubernetes/kubernetes/pull/95985), [@saschagrunert](https://github.com/saschagrunert)) [SIG Auth] -- Fixed bug in reflector that couldn't recover from "Too large resource version" errors with API servers 1.17.0-1.18.5 ([#94316](https://github.com/kubernetes/kubernetes/pull/94316), [@janeczku](https://github.com/janeczku)) [SIG API Machinery] -- Fixed bug where kubectl top pod output is not sorted when --sort-by and --containers flags are used together ([#93692](https://github.com/kubernetes/kubernetes/pull/93692), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] -- Fixed kubelet creating extra sandbox for pods with RestartPolicyOnFailure after all containers succeeded ([#92614](https://github.com/kubernetes/kubernetes/pull/92614), [@tnqn](https://github.com/tnqn)) [SIG Node and Testing] -- Fixes an issue proxying to ipv6 pods without specifying a port ([#94834](https://github.com/kubernetes/kubernetes/pull/94834), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Network] -- Fixes code generation for non-namespaced create subresources fake client test. ([#96586](https://github.com/kubernetes/kubernetes/pull/96586), [@Doude](https://github.com/Doude)) [SIG API Machinery] -- Fixes high CPU usage in kubectl drain ([#95260](https://github.com/kubernetes/kubernetes/pull/95260), [@amandahla](https://github.com/amandahla)) [SIG CLI] -- For vSphere Cloud Provider, If VM of worker node is deleted, the node will also be deleted by node controller ([#92608](https://github.com/kubernetes/kubernetes/pull/92608), [@lubronzhan](https://github.com/lubronzhan)) [SIG Cloud Provider] -- Gracefully delete nodes when their parent scale set went missing ([#95289](https://github.com/kubernetes/kubernetes/pull/95289), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] -- HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. ([#95981](https://github.com/kubernetes/kubernetes/pull/95981), [@caesarxuchao](https://github.com/caesarxuchao)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node] -- If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400. - - If the user specifies a timeout in the request URL that exceeds the maximum request deadline allowed by the apiserver, the request will be aborted with an HTTP 400. ([#96061](https://github.com/kubernetes/kubernetes/pull/96061), [@tkashem](https://github.com/tkashem)) [SIG API Machinery, Network and Testing] -- If we set SelectPolicy MinPolicySelect on scaleUp behavior or scaleDown behavior,Horizontal Pod Autoscaler doesn`t automatically scale the number of pods correctly ([#95647](https://github.com/kubernetes/kubernetes/pull/95647), [@JoshuaAndrew](https://github.com/JoshuaAndrew)) [SIG Apps and Autoscaling] -- Ignore apparmor for non-linux operating systems ([#93220](https://github.com/kubernetes/kubernetes/pull/93220), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] -- Ignore root user check when windows pod starts ([#92355](https://github.com/kubernetes/kubernetes/pull/92355), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] -- Improve error messages related to nodePort endpoint changes conntrack entries cleanup. ([#96251](https://github.com/kubernetes/kubernetes/pull/96251), [@ravens](https://github.com/ravens)) [SIG Network] -- In dual-stack clusters, kubelet will now set up both IPv4 and IPv6 iptables rules, which may - fix some problems, eg with HostPorts. ([#94474](https://github.com/kubernetes/kubernetes/pull/94474), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] -- Increase maximum IOPS of AWS EBS io1 volume to current maximum (64,000). ([#90014](https://github.com/kubernetes/kubernetes/pull/90014), [@jacobmarble](https://github.com/jacobmarble)) -- Ipvs: ensure selected scheduler kernel modules are loaded ([#93040](https://github.com/kubernetes/kubernetes/pull/93040), [@cmluciano](https://github.com/cmluciano)) [SIG Network] -- K8s.io/apimachinery: runtime.DefaultUnstructuredConverter.FromUnstructured now handles converting integer fields to typed float values ([#93250](https://github.com/kubernetes/kubernetes/pull/93250), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] -- Kube-proxy now trims extra spaces found in loadBalancerSourceRanges to match Service validation. ([#94107](https://github.com/kubernetes/kubernetes/pull/94107), [@robscott](https://github.com/robscott)) [SIG Network] -- Kubeadm ensures "kubeadm reset" does not unmount the root "/var/lib/kubelet" directory if it is mounted by the user. ([#93702](https://github.com/kubernetes/kubernetes/pull/93702), [@thtanaka](https://github.com/thtanaka)) -- Kubeadm now makes sure the etcd manifest is regenerated upon upgrade even when no etcd version change takes place ([#94395](https://github.com/kubernetes/kubernetes/pull/94395), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] -- Kubeadm now warns (instead of error out) on missing "ca.key" files for root CA, front-proxy CA and etcd CA, during "kubeadm join --control-plane" if the user has provided all certificates, keys and kubeconfig files which require signing with the given CA keys. ([#94988](https://github.com/kubernetes/kubernetes/pull/94988), [@neolit123](https://github.com/neolit123)) -- Kubeadm: add missing "--experimental-patches" flag to "kubeadm init phase control-plane" ([#95786](https://github.com/kubernetes/kubernetes/pull/95786), [@Sh4d1](https://github.com/Sh4d1)) [SIG Cluster Lifecycle] -- Kubeadm: avoid a panic when determining if the running version of CoreDNS is supported during upgrades ([#94299](https://github.com/kubernetes/kubernetes/pull/94299), [@zouyee](https://github.com/zouyee)) [SIG Cluster Lifecycle] -- Kubeadm: ensure the etcd data directory is created with 0700 permissions during control-plane init and join ([#94102](https://github.com/kubernetes/kubernetes/pull/94102), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade ([#96907](https://github.com/kubernetes/kubernetes/pull/96907), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] -- Kubeadm: fix the bug that kubeadm tries to call 'docker info' even if the CRI socket was for another CR ([#94555](https://github.com/kubernetes/kubernetes/pull/94555), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] -- Kubeadm: for Docker as the container runtime, make the "kubeadm reset" command stop containers before removing them ([#94586](https://github.com/kubernetes/kubernetes/pull/94586), [@BedivereZero](https://github.com/BedivereZero)) [SIG Cluster Lifecycle] -- Kubeadm: make the kubeconfig files for the kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. This makes kubeadm clusters more reseliant to version skew problems during immutable upgrades: https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-controller-manager-kube-scheduler-and-cloud-controller-manager ([#94398](https://github.com/kubernetes/kubernetes/pull/94398), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: relax the validation of kubeconfig server URLs. Allow the user to define custom kubeconfig server URLs without erroring out during validation of existing kubeconfig files (e.g. when using external CA mode). ([#94816](https://github.com/kubernetes/kubernetes/pull/94816), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubectl: print error if users place flags before plugin name ([#92343](https://github.com/kubernetes/kubernetes/pull/92343), [@knight42](https://github.com/knight42)) [SIG CLI] -- Kubelet: assume that swap is disabled when `/proc/swaps` does not exist ([#93931](https://github.com/kubernetes/kubernetes/pull/93931), [@SataQiu](https://github.com/SataQiu)) [SIG Node] -- New Azure instance types do now have correct max data disk count information. ([#94340](https://github.com/kubernetes/kubernetes/pull/94340), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Cloud Provider and Storage] -- Port mapping now allows the same `containerPort` of different containers to different `hostPort` without naming the mapping explicitly. ([#94494](https://github.com/kubernetes/kubernetes/pull/94494), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) -- Print go stack traces at -v=4 and not -v=2 ([#94663](https://github.com/kubernetes/kubernetes/pull/94663), [@soltysh](https://github.com/soltysh)) [SIG CLI] -- Recreate EndpointSlices on rapid Service creation. ([#94730](https://github.com/kubernetes/kubernetes/pull/94730), [@robscott](https://github.com/robscott)) -- Reduce volume name length for vsphere volumes ([#96533](https://github.com/kubernetes/kubernetes/pull/96533), [@gnufied](https://github.com/gnufied)) [SIG Storage] -- Remove ready file and its directory (which is created during volume SetUp) during emptyDir volume TearDown. ([#95770](https://github.com/kubernetes/kubernetes/pull/95770), [@jingxu97](https://github.com/jingxu97)) [SIG Storage] -- Reorganized iptables rules to fix a performance issue ([#95252](https://github.com/kubernetes/kubernetes/pull/95252), [@tssurya](https://github.com/tssurya)) [SIG Network] -- Require feature flag CustomCPUCFSQuotaPeriod if setting a non-default cpuCFSQuotaPeriod in kubelet config. ([#94687](https://github.com/kubernetes/kubernetes/pull/94687), [@karan](https://github.com/karan)) [SIG Node] -- Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. ([#96810](https://github.com/kubernetes/kubernetes/pull/96810), [@liggitt](https://github.com/liggitt)) [SIG Node] -- Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of `OwnerRefInvalidNamespace` are recorded when namespace mismatches between child and owner objects are detected. The [kubectl-check-ownerreferences](https://github.com/kubernetes-sigs/kubectl-check-ownerreferences) tool can be run prior to upgrading to locate existing objects with invalid ownerReferences. - - A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted. - - A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. ([#92743](https://github.com/kubernetes/kubernetes/pull/92743), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Testing] -- Skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:162]: Driver azure-disk doesn't support snapshot type DynamicSnapshot -- skipping - skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:185]: Driver azure-disk doesn't support ntfs -- skipping ([#96144](https://github.com/kubernetes/kubernetes/pull/96144), [@qinpingli](https://github.com/qinpingli)) [SIG Storage and Testing] -- StatefulSet Controller now waits for PersistentVolumeClaim deletion before creating pods. ([#93457](https://github.com/kubernetes/kubernetes/pull/93457), [@ymmt2005](https://github.com/ymmt2005)) -- StreamWatcher now calls HandleCrash at appropriate sequence. ([#93108](https://github.com/kubernetes/kubernetes/pull/93108), [@lixiaobing1](https://github.com/lixiaobing1)) -- Support the node label `node.kubernetes.io/exclude-from-external-load-balancers` ([#95542](https://github.com/kubernetes/kubernetes/pull/95542), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- The AWS network load balancer attributes can now be specified during service creation ([#95247](https://github.com/kubernetes/kubernetes/pull/95247), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] -- The `/debug/api_priority_and_fairness/dump_requests` path at an apiserver will no longer return a phantom line for each exempt priority level. ([#93406](https://github.com/kubernetes/kubernetes/pull/93406), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] -- The kube-apiserver will no longer serve APIs that should have been deleted in GA non-alpha levels. Alpha levels will continue to serve the removed APIs so that CI doesn't immediately break. ([#96525](https://github.com/kubernetes/kubernetes/pull/96525), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] -- The kubelet recognizes the --containerd-namespace flag to configure the namespace used by cadvisor. ([#87054](https://github.com/kubernetes/kubernetes/pull/87054), [@changyaowei](https://github.com/changyaowei)) [SIG Node] -- Unhealthy pods covered by PDBs can be successfully evicted if enough healthy pods are available. ([#94381](https://github.com/kubernetes/kubernetes/pull/94381), [@michaelgugino](https://github.com/michaelgugino)) [SIG Apps] -- Update Calico to v3.15.2 ([#94241](https://github.com/kubernetes/kubernetes/pull/94241), [@lmm](https://github.com/lmm)) [SIG Cloud Provider] -- Update default etcd server version to 3.4.13 ([#94287](https://github.com/kubernetes/kubernetes/pull/94287), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing] -- Update max azure data disk count map ([#96308](https://github.com/kubernetes/kubernetes/pull/96308), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Update the PIP when it is not in the Succeeded provisioning state during the LB update. ([#95748](https://github.com/kubernetes/kubernetes/pull/95748), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Update the frontend IP config when the service's `pipName` annotation is changed ([#95813](https://github.com/kubernetes/kubernetes/pull/95813), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Update the route table tag in the route reconcile loop ([#96545](https://github.com/kubernetes/kubernetes/pull/96545), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Use NLB Subnet CIDRs instead of VPC CIDRs in Health Check SG Rules ([#93515](https://github.com/kubernetes/kubernetes/pull/93515), [@t0rr3sp3dr0](https://github.com/t0rr3sp3dr0)) [SIG Cloud Provider] -- Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. ([#92817](https://github.com/kubernetes/kubernetes/pull/92817), [@kmala](https://github.com/kmala)) [SIG Node] -- Very large patches may now be specified to `kubectl patch` with the `--patch-file` flag instead of including them directly on the command line. The `--patch` and `--patch-file` flags are mutually exclusive. ([#93548](https://github.com/kubernetes/kubernetes/pull/93548), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] -- Volume binding: report UnschedulableAndUnresolvable status instead of an error when bound PVs not found ([#95541](https://github.com/kubernetes/kubernetes/pull/95541), [@cofyc](https://github.com/cofyc)) [SIG Apps, Scheduling and Storage] -- Warn instead of fail when creating Roles and ClusterRoles with custom verbs via kubectl ([#92492](https://github.com/kubernetes/kubernetes/pull/92492), [@eddiezane](https://github.com/eddiezane)) [SIG CLI] -- When creating a PVC with the volume.beta.kubernetes.io/storage-provisioner annotation already set, the PV controller might have incorrectly deleted the newly provisioned PV instead of binding it to the PVC, depending on timing and system load. ([#95909](https://github.com/kubernetes/kubernetes/pull/95909), [@pohly](https://github.com/pohly)) [SIG Apps and Storage] -- [kubectl] Fail when local source file doesn't exist ([#90333](https://github.com/kubernetes/kubernetes/pull/90333), [@bamarni](https://github.com/bamarni)) [SIG CLI] - -### Other (Cleanup or Flake) - -- **Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.**: - - ([#96443](https://github.com/kubernetes/kubernetes/pull/96443), [@alaypatel07](https://github.com/alaypatel07)) [SIG Apps] -- --redirect-container-streaming is no longer functional. The flag will be removed in v1.22 ([#95935](https://github.com/kubernetes/kubernetes/pull/95935), [@tallclair](https://github.com/tallclair)) [SIG Node] -- A new metric `requestAbortsTotal` has been introduced that counts aborted requests for each `group`, `version`, `verb`, `resource`, `subresource` and `scope`. ([#95002](https://github.com/kubernetes/kubernetes/pull/95002), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery, Cloud Provider, Instrumentation and Scheduling] -- API priority and fairness metrics use snake_case in label names ([#96236](https://github.com/kubernetes/kubernetes/pull/96236), [@adtac](https://github.com/adtac)) [SIG API Machinery, Cluster Lifecycle, Instrumentation and Testing] -- Add fine grained debugging to intra-pod conformance test to troubleshoot networking issues for potentially unhealthy nodes when running conformance or sonobuoy tests. ([#93837](https://github.com/kubernetes/kubernetes/pull/93837), [@jayunit100](https://github.com/jayunit100)) -- Add the following metrics: - - network_plugin_operations_total - - network_plugin_operations_errors_total ([#93066](https://github.com/kubernetes/kubernetes/pull/93066), [@AnishShah](https://github.com/AnishShah)) -- Adds a bootstrapping ClusterRole, ClusterRoleBinding and group for /metrics, /livez/*, /readyz/*, & /healthz/- endpoints. ([#93311](https://github.com/kubernetes/kubernetes/pull/93311), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Auth, Cloud Provider and Instrumentation] -- AdmissionReview objects sent for the creation of Namespace API objects now populate the `namespace` attribute consistently (previously the `namespace` attribute was empty for Namespace creation via POST requests, and populated for Namespace creation via server-side-apply PATCH requests) ([#95012](https://github.com/kubernetes/kubernetes/pull/95012), [@nodo](https://github.com/nodo)) [SIG API Machinery and Testing] -- Applies translations on all command descriptions ([#95439](https://github.com/kubernetes/kubernetes/pull/95439), [@HerrNaN](https://github.com/HerrNaN)) [SIG CLI] -- Base-images: Update to debian-iptables:buster-v1.3.0 +- A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. ([#98892](https://github.com/kubernetes/kubernetes/pull/98892), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation] +- A new histogram metric to track the time it took to delete a job by the `TTLAfterFinished` controller ([#98676](https://github.com/kubernetes/kubernetes/pull/98676), [@ahg-g](https://github.com/ahg-g)) +- AWS cloud provider supports auto-discovering subnets without any `kubernetes.io/cluster/` tags. It also supports additional service annotation `service.beta.kubernetes.io/aws-load-balancer-subnets` to manually configure the subnets. ([#97431](https://github.com/kubernetes/kubernetes/pull/97431), [@kishorj](https://github.com/kishorj)) +- Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. ([#98203](https://github.com/kubernetes/kubernetes/pull/98203), [@yuzhiquan](https://github.com/yuzhiquan)) +- Add --permit-address-sharing flag to `kube-apiserver` to listen with `SO_REUSEADDR`. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoids waiting for the kernel to release socket in `TIME_WAIT` state, and hence, considerably reducing `kube-apiserver` restart times under certain conditions. ([#93861](https://github.com/kubernetes/kubernetes/pull/93861), [@sttts](https://github.com/sttts)) +- Add `csi_operations_seconds` metric on kubelet that exposes CSI operations duration and status for node CSI operations. ([#98979](https://github.com/kubernetes/kubernetes/pull/98979), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Instrumentation and Storage] +- Add `migrated` field into `storage_operation_duration_seconds` metric ([#99050](https://github.com/kubernetes/kubernetes/pull/99050), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Instrumentation and Storage] +- Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. ([#97009](https://github.com/kubernetes/kubernetes/pull/97009), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery and Scalability] +- Add metric etcd_lease_object_counts for kube-apiserver to observe max objects attached to a single etcd lease. ([#97480](https://github.com/kubernetes/kubernetes/pull/97480), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery, Instrumentation and Scalability] +- Add support to generate client-side binaries for new darwin/arm64 platform ([#97743](https://github.com/kubernetes/kubernetes/pull/97743), [@dims](https://github.com/dims)) [SIG Release and Testing] +- Added `ephemeral_volume_controller_create[_failures]_total` counters to kube-controller-manager metrics ([#99115](https://github.com/kubernetes/kubernetes/pull/99115), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage] +- Added support for installing `arm64` node artifacts. ([#99242](https://github.com/kubernetes/kubernetes/pull/99242), [@liu-cong](https://github.com/liu-cong)) +- Adds alpha feature `VolumeCapacityPriority` which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. ([#96347](https://github.com/kubernetes/kubernetes/pull/96347), [@cofyc](https://github.com/cofyc)) [SIG Apps, Network, Scheduling, Storage and Testing] +- Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. ([#96502](https://github.com/kubernetes/kubernetes/pull/96502), [@249043822](https://github.com/249043822)) [SIG Auth] +- Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, as well as a gauge for the missed schedules of a cronjob ([#99341](https://github.com/kubernetes/kubernetes/pull/99341), [@alaypatel07](https://github.com/alaypatel07)) +- Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. ([#98952](https://github.com/kubernetes/kubernetes/pull/98952), [@seans3](https://github.com/seans3)) +- Base-images: Update to debian-iptables:buster-v1.4.0 - Uses iptables 1.8.5 - - base-images: Update to debian-base:buster-v1.2.0 - - cluster/images/etcd: Build etcd:3.4.13-1 image - - Uses debian-base:buster-v1.2.0 ([#94733](https://github.com/kubernetes/kubernetes/pull/94733), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release and Testing] -- Changed: default "Accept-Encoding" header removed from HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes ([#96127](https://github.com/kubernetes/kubernetes/pull/96127), [@fonsecas72](https://github.com/fonsecas72)) [SIG Network and Node] -- Client-go header logging (at verbosity levels >= 9) now masks `Authorization` header contents ([#95316](https://github.com/kubernetes/kubernetes/pull/95316), [@sfowl](https://github.com/sfowl)) [SIG API Machinery] -- Decrease warning message frequency on setting volume ownership for configmap/secret. ([#92878](https://github.com/kubernetes/kubernetes/pull/92878), [@jvanz](https://github.com/jvanz)) -- Enhance log information of verifyRunAsNonRoot, add pod, container information ([#94911](https://github.com/kubernetes/kubernetes/pull/94911), [@wawa0210](https://github.com/wawa0210)) [SIG Node] -- Fix func name NewCreateCreateDeploymentOptions ([#91931](https://github.com/kubernetes/kubernetes/pull/91931), [@lixiaobing1](https://github.com/lixiaobing1)) [SIG CLI] -- Fix kubelet to properly log when a container is started. Previously, kubelet may log that container is dead and was restarted when it was actually started for the first time. This behavior only happened on pods with initContainers and regular containers. ([#91469](https://github.com/kubernetes/kubernetes/pull/91469), [@rata](https://github.com/rata)) -- Fixes the message about no auth for metrics in scheduler. ([#94035](https://github.com/kubernetes/kubernetes/pull/94035), [@zhouya0](https://github.com/zhouya0)) [SIG Scheduling] -- Generators for services are removed from kubectl ([#95256](https://github.com/kubernetes/kubernetes/pull/95256), [@Git-Jiro](https://github.com/Git-Jiro)) [SIG CLI] -- Introduce kubectl-convert plugin. ([#96190](https://github.com/kubernetes/kubernetes/pull/96190), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing] -- Kube-scheduler now logs processed component config at startup ([#96426](https://github.com/kubernetes/kubernetes/pull/96426), [@damemi](https://github.com/damemi)) [SIG Scheduling] -- Kubeadm: Separate argument key/value in log msg ([#94016](https://github.com/kubernetes/kubernetes/pull/94016), [@mrueg](https://github.com/mrueg)) [SIG Cluster Lifecycle] -- Kubeadm: remove the CoreDNS check for known image digests when applying the addon ([#94506](https://github.com/kubernetes/kubernetes/pull/94506), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 ([#95419](https://github.com/kubernetes/kubernetes/pull/95419), [@jsturtevant](https://github.com/jsturtevant)) [SIG Cluster Lifecycle and Windows] -- Kubectl: the `generator` flag of `kubectl autoscale` has been deprecated and has no effect, it will be removed in a feature release ([#92998](https://github.com/kubernetes/kubernetes/pull/92998), [@SataQiu](https://github.com/SataQiu)) [SIG CLI] -- Lock ExternalPolicyForExternalIP to default, this feature gate will be removed in 1.22. ([#94581](https://github.com/kubernetes/kubernetes/pull/94581), [@knabben](https://github.com/knabben)) [SIG Network] -- Mask ceph RBD adminSecrets in logs when logLevel >= 4. ([#95245](https://github.com/kubernetes/kubernetes/pull/95245), [@sfowl](https://github.com/sfowl)) -- Remove offensive words from kubectl cluster-info command. ([#95202](https://github.com/kubernetes/kubernetes/pull/95202), [@rikatz](https://github.com/rikatz)) -- Remove support for "ci/k8s-master" version label in kubeadm, use "ci/latest" instead. See [kubernetes/test-infra#18517](https://github.com/kubernetes/test-infra/pull/18517). ([#93626](https://github.com/kubernetes/kubernetes/pull/93626), [@vikkyomkar](https://github.com/vikkyomkar)) -- Remove the dependency of csi-translation-lib module on apiserver/cloud-provider/controller-manager ([#95543](https://github.com/kubernetes/kubernetes/pull/95543), [@wawa0210](https://github.com/wawa0210)) [SIG Release] -- Scheduler framework interface moved from pkg/scheduler/framework/v1alpha to pkg/scheduler/framework ([#95069](https://github.com/kubernetes/kubernetes/pull/95069), [@farah](https://github.com/farah)) [SIG Scheduling, Storage and Testing] -- Service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset is removed. All Standard load balancers will always enable tcp resets. ([#94297](https://github.com/kubernetes/kubernetes/pull/94297), [@MarcPow](https://github.com/MarcPow)) [SIG Cloud Provider] -- Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver ([#94397](https://github.com/kubernetes/kubernetes/pull/94397), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] -- Strip unnecessary security contexts on Windows ([#93475](https://github.com/kubernetes/kubernetes/pull/93475), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Node, Testing and Windows] -- To ensure the code be strong, add unit test for GetAddressAndDialer ([#93180](https://github.com/kubernetes/kubernetes/pull/93180), [@FreeZhang61](https://github.com/FreeZhang61)) [SIG Node] -- UDP and SCTP protocols can left stale connections that need to be cleared to avoid services disruption, but they can cause problems that are hard to debug. - Kubernetes components using a loglevel greater or equal than 4 will log the conntrack operations and its output, to show the entries that were deleted. ([#95694](https://github.com/kubernetes/kubernetes/pull/95694), [@aojea](https://github.com/aojea)) [SIG Network] -- Update CNI plugins to v0.8.7 ([#94367](https://github.com/kubernetes/kubernetes/pull/94367), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Network, Node, Release and Testing] -- Update cri-tools to [v1.19.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.19.0) ([#94307](https://github.com/kubernetes/kubernetes/pull/94307), [@xmudrii](https://github.com/xmudrii)) [SIG Cloud Provider] -- Update etcd client side to v3.4.13 ([#94259](https://github.com/kubernetes/kubernetes/pull/94259), [@jingyih](https://github.com/jingyih)) [SIG API Machinery and Cloud Provider] -- Users will now be able to configure all supported values for AWS NLB health check interval and thresholds for new resources. ([#96312](https://github.com/kubernetes/kubernetes/pull/96312), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] -- V1helpers.MatchNodeSelectorTerms now accepts just a Node and a list of Terms ([#95871](https://github.com/kubernetes/kubernetes/pull/95871), [@damemi](https://github.com/damemi)) [SIG Apps, Scheduling and Storage] -- Vsphere: improve logging message on node cache refresh event ([#95236](https://github.com/kubernetes/kubernetes/pull/95236), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] -- `MatchNodeSelectorTerms` function moved to `k8s.io/component-helpers` ([#95531](https://github.com/kubernetes/kubernetes/pull/95531), [@damemi](https://github.com/damemi)) [SIG Apps, Scheduling and Storage] -- `kubectl api-resources` now prints the API version (as 'API group/version', same as output of `kubectl api-versions`). The column APIGROUP is now APIVERSION ([#95253](https://github.com/kubernetes/kubernetes/pull/95253), [@sallyom](https://github.com/sallyom)) [SIG CLI] -- `kubectl get ingress` now prefers the `networking.k8s.io/v1` over `extensions/v1beta1` (deprecated since v1.14). To explicitly request the deprecated version, use `kubectl get ingress.v1beta1.extensions`. ([#94309](https://github.com/kubernetes/kubernetes/pull/94309), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and CLI] + - base-images: Update to debian-base:buster-v1.3.0 + - cluster/images/etcd: Build etcd:3.4.13-2 image + - Uses debian-base:buster-v1.3.0 ([#98401](https://github.com/kubernetes/kubernetes/pull/98401), [@pacoxu](https://github.com/pacoxu)) [SIG Testing] +- CRIContainerLogRotation graduates to GA and unconditionally enabled. ([#99651](https://github.com/kubernetes/kubernetes/pull/99651), [@umohnani8](https://github.com/umohnani8)) +- Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. ([#99385](https://github.com/kubernetes/kubernetes/pull/99385), [@YoyinZyc](https://github.com/YoyinZyc)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Release] +- Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. ([#99738](https://github.com/kubernetes/kubernetes/pull/99738), [@YoyinZyc](https://github.com/YoyinZyc)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] +- EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. ([#100319](https://github.com/kubernetes/kubernetes/pull/100319), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] +- Enables Kubelet to check volume condition and log events to corresponding pods. ([#99284](https://github.com/kubernetes/kubernetes/pull/99284), [@fengzixu](https://github.com/fengzixu)) [SIG Apps, Instrumentation, Node and Storage] +- EndpointSliceNodeName graduates to GA and thus will be unconditionally enabled -- NodeName will always be available in the v1beta1 API. ([#99746](https://github.com/kubernetes/kubernetes/pull/99746), [@swetharepakula](https://github.com/swetharepakula)) +- Export `NewDebuggingRoundTripper` function and `DebugLevel` options in the k8s.io/client-go/transport package. ([#98324](https://github.com/kubernetes/kubernetes/pull/98324), [@atosatto](https://github.com/atosatto)) +- Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration ([#99653](https://github.com/kubernetes/kubernetes/pull/99653), [@aojea](https://github.com/aojea)) [SIG Instrumentation and Network] +- Kube-scheduler now logs plugin scoring summaries at --v=4 ([#99411](https://github.com/kubernetes/kubernetes/pull/99411), [@damemi](https://github.com/damemi)) [SIG Scheduling] +- Kubeadm now includes CoreDNS v1.8.0. ([#96429](https://github.com/kubernetes/kubernetes/pull/96429), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] +- Kubeadm: IPv6DualStack feature gate graduates to Beta and enabled by default ([#99294](https://github.com/kubernetes/kubernetes/pull/99294), [@pacoxu](https://github.com/pacoxu)) +- Kubeadm: a warning to user as ipv6 site-local is deprecated ([#99574](https://github.com/kubernetes/kubernetes/pull/99574), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle and Network] +- Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. ([#97266](https://github.com/kubernetes/kubernetes/pull/97266), [@robbiemcmichael](https://github.com/robbiemcmichael)) [SIG Cluster Lifecycle] +- Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional ([#96378](https://github.com/kubernetes/kubernetes/pull/96378), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle and Node] +- Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. ([#98269](https://github.com/kubernetes/kubernetes/pull/98269), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. ([#99476](https://github.com/kubernetes/kubernetes/pull/99476), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Kubeadm: perform pre-flight validation on host/node name upon `kubeadm init` and `kubeadm join`, showing warnings on non-compliant names ([#99194](https://github.com/kubernetes/kubernetes/pull/99194), [@pacoxu](https://github.com/pacoxu)) +- Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. ([#98250](https://github.com/kubernetes/kubernetes/pull/98250), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] +- Kubectl: Add `--use-protocol-buffers` flag to kubectl top pods and nodes. ([#96655](https://github.com/kubernetes/kubernetes/pull/96655), [@serathius](https://github.com/serathius)) +- Kubectl: `kubectl get` will omit managed fields by default now. Users could set `--show-managed-fields` to true to show managedFields when the output format is either `json` or `yaml`. ([#96878](https://github.com/kubernetes/kubernetes/pull/96878), [@knight42](https://github.com/knight42)) [SIG CLI and Testing] +- Kubectl: a Pod can be preselected as default container using `kubectl.kubernetes.io/default-container` annotation ([#99833](https://github.com/kubernetes/kubernetes/pull/99833), [@mengjiao-liu](https://github.com/mengjiao-liu)) +- Kubectl: add bash-completion for comma separated list on `kubectl get` ([#98301](https://github.com/kubernetes/kubernetes/pull/98301), [@phil9909](https://github.com/phil9909)) +- Kubernetes is now built using go1.15.8 ([#98834](https://github.com/kubernetes/kubernetes/pull/98834), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] +- Kubernetes is now built with Golang 1.16 ([#98572](https://github.com/kubernetes/kubernetes/pull/98572), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing] +- Kubernetes is now built with Golang 1.16.1 ([#100106](https://github.com/kubernetes/kubernetes/pull/100106), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] +- Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=metric1,metric2') ([#99217](https://github.com/kubernetes/kubernetes/pull/99217), [@logicalhan](https://github.com/logicalhan)) +- New admission controller `DenyServiceExternalIPs` is available. Clusters which do not *need* the Service `externalIPs` feature should enable this controller and be more secure. ([#97395](https://github.com/kubernetes/kubernetes/pull/97395), [@thockin](https://github.com/thockin)) +- Overall, enable the feature of `PreferNominatedNode` will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of `PreferNominatedNode`, the pod might not be scheduled to the best candidate node in the cluster. ([#93179](https://github.com/kubernetes/kubernetes/pull/93179), [@chendave](https://github.com/chendave)) [SIG Scheduling and Testing] +- Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. ([#99361](https://github.com/kubernetes/kubernetes/pull/99361), [@Novex](https://github.com/Novex)) [SIG Storage] +- Port the devicemanager to Windows node to allow device plugins like directx ([#93285](https://github.com/kubernetes/kubernetes/pull/93285), [@aarnaud](https://github.com/aarnaud)) [SIG Node, Testing and Windows] +- Removes cAdvisor JSON metrics (/stats/container, /stats//, /stats////) from the kubelet. ([#99236](https://github.com/kubernetes/kubernetes/pull/99236), [@pacoxu](https://github.com/pacoxu)) +- Rename metrics `etcd_object_counts` to `apiserver_storage_object_counts` and mark it as stable. The original `etcd_object_counts` metrics name is marked as "Deprecated" and will be removed in the future. ([#99785](https://github.com/kubernetes/kubernetes/pull/99785), [@erain](https://github.com/erain)) [SIG API Machinery, Instrumentation and Testing] +- Sysctls graduates to General Availability and thus unconditionally enabled. ([#99158](https://github.com/kubernetes/kubernetes/pull/99158), [@wgahnagl](https://github.com/wgahnagl)) +- The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. ([#97322](https://github.com/kubernetes/kubernetes/pull/97322), [@claudiubelu](https://github.com/claudiubelu)) [SIG Windows] +- The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. ([#99213](https://github.com/kubernetes/kubernetes/pull/99213), [@AliceZhang2016](https://github.com/AliceZhang2016)) [SIG Scheduling] +- The `CronJobControllerV2` feature flag graduates to Beta and set to be enabled by default. ([#98878](https://github.com/kubernetes/kubernetes/pull/98878), [@soltysh](https://github.com/soltysh)) +- The `EndpointSlice` mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. + The well-known annotation `endpoints.kubernetes.io/last-change-trigger-time` is skipped and not mirrored. ([#98116](https://github.com/kubernetes/kubernetes/pull/98116), [@aojea](https://github.com/aojea)) +- The `RunAsGroup` feature has been promoted to GA in this release. ([#94641](https://github.com/kubernetes/kubernetes/pull/94641), [@krmayankk](https://github.com/krmayankk)) [SIG Auth and Node] +- The `ServiceAccountIssuerDiscovery` feature has graduated to GA, and is unconditionally enabled. The `ServiceAccountIssuerDiscovery` feature-gate will be removed in 1.22. ([#98553](https://github.com/kubernetes/kubernetes/pull/98553), [@mtaufen](https://github.com/mtaufen)) [SIG API Machinery, Auth and Testing] +- The `TTLAfterFinished` feature flag is now beta and enabled by default ([#98678](https://github.com/kubernetes/kubernetes/pull/98678), [@ahg-g](https://github.com/ahg-g)) +- The apimachinery util/net function used to detect the bind address `ResolveBindAddress()` takes into consideration global IP addresses on loopback interfaces when 1) the host has default routes, or 2) there are no global IPs on those interfaces in order to support more complex network scenarios like BGP Unnumbered RFC 5549 ([#95790](https://github.com/kubernetes/kubernetes/pull/95790), [@aojea](https://github.com/aojea)) [SIG Network] +- The feature gate `RootCAConfigMap` graduated to GA in v1.21 and therefore will be unconditionally enabled. This flag will be removed in v1.22 release. ([#98033](https://github.com/kubernetes/kubernetes/pull/98033), [@zshihang](https://github.com/zshihang)) +- The pause image upgraded to `v3.4.1` in kubelet and kubeadm for both Linux and Windows. ([#98205](https://github.com/kubernetes/kubernetes/pull/98205), [@pacoxu](https://github.com/pacoxu)) +- Update pause container to run as pseudo user and group `65535:65535`. This implies the release of version 3.5 of the container images. ([#97963](https://github.com/kubernetes/kubernetes/pull/97963), [@saschagrunert](https://github.com/saschagrunert)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing] +- Update the latest validated version of Docker to 20.10 ([#98977](https://github.com/kubernetes/kubernetes/pull/98977), [@neolit123](https://github.com/neolit123)) [SIG CLI, Cluster Lifecycle and Node] +- Upgrade node local dns to 1.17.0 for better IPv6 support ([#99749](https://github.com/kubernetes/kubernetes/pull/99749), [@pacoxu](https://github.com/pacoxu)) [SIG Cloud Provider and Network] +- Upgrades `IPv6Dualstack` to `Beta` and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: [IPv4/IPv6 Dual-stack](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack) ([#98969](https://github.com/kubernetes/kubernetes/pull/98969), [@khenidak](https://github.com/khenidak)) +- Users might specify the `kubectl.kubernetes.io/default-container` annotation in a Pod to preselect container for kubectl commands. ([#99581](https://github.com/kubernetes/kubernetes/pull/99581), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG CLI] +- When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. ([#99212](https://github.com/kubernetes/kubernetes/pull/99212), [@damemi](https://github.com/damemi)) [SIG Apps and Testing] +- When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables + for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU + and memory usage of the kube-apiserver in large clusters. ([#99393](https://github.com/kubernetes/kubernetes/pull/99393), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery, Node and Testing] +- WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. ([#99794](https://github.com/kubernetes/kubernetes/pull/99794), [@robscott](https://github.com/robscott)) [SIG Network] +- `kubectl wait` ensures that observedGeneration >= generation to prevent stale state reporting. An example scenario can be found on CRD updates. ([#97408](https://github.com/kubernetes/kubernetes/pull/97408), [@KnicKnic](https://github.com/KnicKnic)) -## Dependencies +### Documentation -### Added -- cloud.google.com/go/firestore: v1.1.0 -- github.com/Azure/go-autorest: [v14.2.0+incompatible](https://github.com/Azure/go-autorest/tree/v14.2.0) -- github.com/armon/go-metrics: [f0300d1](https://github.com/armon/go-metrics/tree/f0300d1) -- github.com/armon/go-radix: [7fddfc3](https://github.com/armon/go-radix/tree/7fddfc3) -- github.com/bketelsen/crypt: [5cbc8cc](https://github.com/bketelsen/crypt/tree/5cbc8cc) -- github.com/form3tech-oss/jwt-go: [v3.2.2+incompatible](https://github.com/form3tech-oss/jwt-go/tree/v3.2.2) -- github.com/fvbommel/sortorder: [v1.0.1](https://github.com/fvbommel/sortorder/tree/v1.0.1) -- github.com/hashicorp/consul/api: [v1.1.0](https://github.com/hashicorp/consul/api/tree/v1.1.0) -- github.com/hashicorp/consul/sdk: [v0.1.1](https://github.com/hashicorp/consul/sdk/tree/v0.1.1) -- github.com/hashicorp/errwrap: [v1.0.0](https://github.com/hashicorp/errwrap/tree/v1.0.0) -- github.com/hashicorp/go-cleanhttp: [v0.5.1](https://github.com/hashicorp/go-cleanhttp/tree/v0.5.1) -- github.com/hashicorp/go-immutable-radix: [v1.0.0](https://github.com/hashicorp/go-immutable-radix/tree/v1.0.0) -- github.com/hashicorp/go-msgpack: [v0.5.3](https://github.com/hashicorp/go-msgpack/tree/v0.5.3) -- github.com/hashicorp/go-multierror: [v1.0.0](https://github.com/hashicorp/go-multierror/tree/v1.0.0) -- github.com/hashicorp/go-rootcerts: [v1.0.0](https://github.com/hashicorp/go-rootcerts/tree/v1.0.0) -- github.com/hashicorp/go-sockaddr: [v1.0.0](https://github.com/hashicorp/go-sockaddr/tree/v1.0.0) -- github.com/hashicorp/go-uuid: [v1.0.1](https://github.com/hashicorp/go-uuid/tree/v1.0.1) -- github.com/hashicorp/go.net: [v0.0.1](https://github.com/hashicorp/go.net/tree/v0.0.1) -- github.com/hashicorp/logutils: [v1.0.0](https://github.com/hashicorp/logutils/tree/v1.0.0) -- github.com/hashicorp/mdns: [v1.0.0](https://github.com/hashicorp/mdns/tree/v1.0.0) -- github.com/hashicorp/memberlist: [v0.1.3](https://github.com/hashicorp/memberlist/tree/v0.1.3) -- github.com/hashicorp/serf: [v0.8.2](https://github.com/hashicorp/serf/tree/v0.8.2) -- github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) -- github.com/mitchellh/cli: [v1.0.0](https://github.com/mitchellh/cli/tree/v1.0.0) -- github.com/mitchellh/go-testing-interface: [v1.0.0](https://github.com/mitchellh/go-testing-interface/tree/v1.0.0) -- github.com/mitchellh/gox: [v0.4.0](https://github.com/mitchellh/gox/tree/v0.4.0) -- github.com/mitchellh/iochan: [v1.0.0](https://github.com/mitchellh/iochan/tree/v1.0.0) -- github.com/pascaldekloe/goe: [57f6aae](https://github.com/pascaldekloe/goe/tree/57f6aae) -- github.com/posener/complete: [v1.1.1](https://github.com/posener/complete/tree/v1.1.1) -- github.com/ryanuber/columnize: [9b3edd6](https://github.com/ryanuber/columnize/tree/9b3edd6) -- github.com/sean-/seed: [e2103e2](https://github.com/sean-/seed/tree/e2103e2) -- github.com/subosito/gotenv: [v1.2.0](https://github.com/subosito/gotenv/tree/v1.2.0) -- github.com/willf/bitset: [d5bec33](https://github.com/willf/bitset/tree/d5bec33) -- gopkg.in/ini.v1: v1.51.0 -- gopkg.in/yaml.v3: 9f266ea -- rsc.io/quote/v3: v3.1.0 -- rsc.io/sampler: v1.3.0 - -### Changed -- cloud.google.com/go/bigquery: v1.0.1 → v1.4.0 -- cloud.google.com/go/datastore: v1.0.0 → v1.1.0 -- cloud.google.com/go/pubsub: v1.0.1 → v1.2.0 -- cloud.google.com/go/storage: v1.0.0 → v1.6.0 -- cloud.google.com/go: v0.51.0 → v0.54.0 -- github.com/Azure/go-autorest/autorest/adal: [v0.8.2 → v0.9.5](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.8.2...v0.9.5) -- github.com/Azure/go-autorest/autorest/date: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.2.0...v0.3.0) -- github.com/Azure/go-autorest/autorest/mocks: [v0.3.0 → v0.4.1](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.3.0...v0.4.1) -- github.com/Azure/go-autorest/autorest: [v0.9.6 → v0.11.1](https://github.com/Azure/go-autorest/autorest/compare/v0.9.6...v0.11.1) -- github.com/Azure/go-autorest/logger: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/logger/compare/v0.1.0...v0.2.0) -- github.com/Azure/go-autorest/tracing: [v0.5.0 → v0.6.0](https://github.com/Azure/go-autorest/tracing/compare/v0.5.0...v0.6.0) -- github.com/Microsoft/go-winio: [fc70bd9 → v0.4.15](https://github.com/Microsoft/go-winio/compare/fc70bd9...v0.4.15) -- github.com/aws/aws-sdk-go: [v1.28.2 → v1.35.24](https://github.com/aws/aws-sdk-go/compare/v1.28.2...v1.35.24) -- github.com/blang/semver: [v3.5.0+incompatible → v3.5.1+incompatible](https://github.com/blang/semver/compare/v3.5.0...v3.5.1) -- github.com/checkpoint-restore/go-criu/v4: [v4.0.2 → v4.1.0](https://github.com/checkpoint-restore/go-criu/v4/compare/v4.0.2...v4.1.0) -- github.com/containerd/containerd: [v1.3.3 → v1.4.1](https://github.com/containerd/containerd/compare/v1.3.3...v1.4.1) -- github.com/containerd/ttrpc: [v1.0.0 → v1.0.2](https://github.com/containerd/ttrpc/compare/v1.0.0...v1.0.2) -- github.com/containerd/typeurl: [v1.0.0 → v1.0.1](https://github.com/containerd/typeurl/compare/v1.0.0...v1.0.1) -- github.com/coreos/etcd: [v3.3.10+incompatible → v3.3.13+incompatible](https://github.com/coreos/etcd/compare/v3.3.10...v3.3.13) -- github.com/docker/docker: [aa6a989 → bd33bbf](https://github.com/docker/docker/compare/aa6a989...bd33bbf) -- github.com/go-gl/glfw/v3.3/glfw: [12ad95a → 6f7a984](https://github.com/go-gl/glfw/v3.3/glfw/compare/12ad95a...6f7a984) -- github.com/golang/groupcache: [215e871 → 8c9f03a](https://github.com/golang/groupcache/compare/215e871...8c9f03a) -- github.com/golang/mock: [v1.3.1 → v1.4.1](https://github.com/golang/mock/compare/v1.3.1...v1.4.1) -- github.com/golang/protobuf: [v1.4.2 → v1.4.3](https://github.com/golang/protobuf/compare/v1.4.2...v1.4.3) -- github.com/google/cadvisor: [v0.37.0 → v0.38.5](https://github.com/google/cadvisor/compare/v0.37.0...v0.38.5) -- github.com/google/go-cmp: [v0.4.0 → v0.5.2](https://github.com/google/go-cmp/compare/v0.4.0...v0.5.2) -- github.com/google/pprof: [d4f498a → 1ebb73c](https://github.com/google/pprof/compare/d4f498a...1ebb73c) -- github.com/google/uuid: [v1.1.1 → v1.1.2](https://github.com/google/uuid/compare/v1.1.1...v1.1.2) -- github.com/gorilla/mux: [v1.7.3 → v1.8.0](https://github.com/gorilla/mux/compare/v1.7.3...v1.8.0) -- github.com/gorilla/websocket: [v1.4.0 → v1.4.2](https://github.com/gorilla/websocket/compare/v1.4.0...v1.4.2) -- github.com/jmespath/go-jmespath: [c2b33e8 → v0.4.0](https://github.com/jmespath/go-jmespath/compare/c2b33e8...v0.4.0) -- github.com/karrick/godirwalk: [v1.7.5 → v1.16.1](https://github.com/karrick/godirwalk/compare/v1.7.5...v1.16.1) -- github.com/opencontainers/go-digest: [v1.0.0-rc1 → v1.0.0](https://github.com/opencontainers/go-digest/compare/v1.0.0-rc1...v1.0.0) -- github.com/opencontainers/runc: [819fcc6 → v1.0.0-rc92](https://github.com/opencontainers/runc/compare/819fcc6...v1.0.0-rc92) -- github.com/opencontainers/runtime-spec: [237cc4f → 4d89ac9](https://github.com/opencontainers/runtime-spec/compare/237cc4f...4d89ac9) -- github.com/opencontainers/selinux: [v1.5.2 → v1.6.0](https://github.com/opencontainers/selinux/compare/v1.5.2...v1.6.0) -- github.com/prometheus/procfs: [v0.1.3 → v0.2.0](https://github.com/prometheus/procfs/compare/v0.1.3...v0.2.0) -- github.com/quobyte/api: [v0.1.2 → v0.1.8](https://github.com/quobyte/api/compare/v0.1.2...v0.1.8) -- github.com/spf13/cobra: [v1.0.0 → v1.1.1](https://github.com/spf13/cobra/compare/v1.0.0...v1.1.1) -- github.com/spf13/viper: [v1.4.0 → v1.7.0](https://github.com/spf13/viper/compare/v1.4.0...v1.7.0) -- github.com/storageos/go-api: [343b3ef → v2.2.0+incompatible](https://github.com/storageos/go-api/compare/343b3ef...v2.2.0) -- github.com/stretchr/testify: [v1.4.0 → v1.6.1](https://github.com/stretchr/testify/compare/v1.4.0...v1.6.1) -- github.com/vishvananda/netns: [52d707b → db3c7e5](https://github.com/vishvananda/netns/compare/52d707b...db3c7e5) -- go.etcd.io/etcd: 17cef6e → dd1b699 -- go.opencensus.io: v0.22.2 → v0.22.3 -- golang.org/x/crypto: 75b2880 → 7f63de1 -- golang.org/x/exp: da58074 → 6cc2880 -- golang.org/x/lint: fdd1cda → 738671d -- golang.org/x/net: ab34263 → 69a7880 -- golang.org/x/oauth2: 858c2ad → bf48bf1 -- golang.org/x/sys: ed371f2 → 5cba982 -- golang.org/x/text: v0.3.3 → v0.3.4 -- golang.org/x/time: 555d28b → 3af7569 -- golang.org/x/xerrors: 9bdfabe → 5ec99f8 -- google.golang.org/api: v0.15.1 → v0.20.0 -- google.golang.org/genproto: cb27e3a → 8816d57 -- google.golang.org/grpc: v1.27.0 → v1.27.1 -- google.golang.org/protobuf: v1.24.0 → v1.25.0 -- honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3 -- k8s.io/gengo: 8167cfd → 83324d8 -- k8s.io/klog/v2: v2.2.0 → v2.4.0 -- k8s.io/kube-openapi: 6aeccd4 → d219536 -- k8s.io/system-validators: v1.1.2 → v1.2.0 -- k8s.io/utils: d5654de → 67b214c -- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.14 -- sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → v4.0.2 - -### Removed -- github.com/armon/consul-api: [eb2c6b5](https://github.com/armon/consul-api/tree/eb2c6b5) -- github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) -- github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) -- github.com/xlab/handysort: [fb3537e](https://github.com/xlab/handysort/tree/fb3537e) -- github.com/xordataexchange/crypt: [b2862e3](https://github.com/xordataexchange/crypt/tree/b2862e3) -- vbom.ml/util: db5cfe1 - - -## Dependencies - -### Added -- cloud.google.com/go/firestore: v1.1.0 -- github.com/Azure/go-autorest: [v14.2.0+incompatible](https://github.com/Azure/go-autorest/tree/v14.2.0) -- github.com/armon/go-metrics: [f0300d1](https://github.com/armon/go-metrics/tree/f0300d1) -- github.com/armon/go-radix: [7fddfc3](https://github.com/armon/go-radix/tree/7fddfc3) -- github.com/bketelsen/crypt: [5cbc8cc](https://github.com/bketelsen/crypt/tree/5cbc8cc) -- github.com/form3tech-oss/jwt-go: [v3.2.2+incompatible](https://github.com/form3tech-oss/jwt-go/tree/v3.2.2) -- github.com/fvbommel/sortorder: [v1.0.1](https://github.com/fvbommel/sortorder/tree/v1.0.1) -- github.com/hashicorp/consul/api: [v1.1.0](https://github.com/hashicorp/consul/api/tree/v1.1.0) -- github.com/hashicorp/consul/sdk: [v0.1.1](https://github.com/hashicorp/consul/sdk/tree/v0.1.1) -- github.com/hashicorp/errwrap: [v1.0.0](https://github.com/hashicorp/errwrap/tree/v1.0.0) -- github.com/hashicorp/go-cleanhttp: [v0.5.1](https://github.com/hashicorp/go-cleanhttp/tree/v0.5.1) -- github.com/hashicorp/go-immutable-radix: [v1.0.0](https://github.com/hashicorp/go-immutable-radix/tree/v1.0.0) -- github.com/hashicorp/go-msgpack: [v0.5.3](https://github.com/hashicorp/go-msgpack/tree/v0.5.3) -- github.com/hashicorp/go-multierror: [v1.0.0](https://github.com/hashicorp/go-multierror/tree/v1.0.0) -- github.com/hashicorp/go-rootcerts: [v1.0.0](https://github.com/hashicorp/go-rootcerts/tree/v1.0.0) -- github.com/hashicorp/go-sockaddr: [v1.0.0](https://github.com/hashicorp/go-sockaddr/tree/v1.0.0) -- github.com/hashicorp/go-uuid: [v1.0.1](https://github.com/hashicorp/go-uuid/tree/v1.0.1) -- github.com/hashicorp/go.net: [v0.0.1](https://github.com/hashicorp/go.net/tree/v0.0.1) -- github.com/hashicorp/logutils: [v1.0.0](https://github.com/hashicorp/logutils/tree/v1.0.0) -- github.com/hashicorp/mdns: [v1.0.0](https://github.com/hashicorp/mdns/tree/v1.0.0) -- github.com/hashicorp/memberlist: [v0.1.3](https://github.com/hashicorp/memberlist/tree/v0.1.3) -- github.com/hashicorp/serf: [v0.8.2](https://github.com/hashicorp/serf/tree/v0.8.2) -- github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) -- github.com/mitchellh/cli: [v1.0.0](https://github.com/mitchellh/cli/tree/v1.0.0) -- github.com/mitchellh/go-testing-interface: [v1.0.0](https://github.com/mitchellh/go-testing-interface/tree/v1.0.0) -- github.com/mitchellh/gox: [v0.4.0](https://github.com/mitchellh/gox/tree/v0.4.0) -- github.com/mitchellh/iochan: [v1.0.0](https://github.com/mitchellh/iochan/tree/v1.0.0) -- github.com/pascaldekloe/goe: [57f6aae](https://github.com/pascaldekloe/goe/tree/57f6aae) -- github.com/posener/complete: [v1.1.1](https://github.com/posener/complete/tree/v1.1.1) -- github.com/ryanuber/columnize: [9b3edd6](https://github.com/ryanuber/columnize/tree/9b3edd6) -- github.com/sean-/seed: [e2103e2](https://github.com/sean-/seed/tree/e2103e2) -- github.com/subosito/gotenv: [v1.2.0](https://github.com/subosito/gotenv/tree/v1.2.0) -- github.com/willf/bitset: [d5bec33](https://github.com/willf/bitset/tree/d5bec33) -- gopkg.in/ini.v1: v1.51.0 -- gopkg.in/yaml.v3: 9f266ea -- rsc.io/quote/v3: v3.1.0 -- rsc.io/sampler: v1.3.0 - -### Changed -- cloud.google.com/go/bigquery: v1.0.1 → v1.4.0 -- cloud.google.com/go/datastore: v1.0.0 → v1.1.0 -- cloud.google.com/go/pubsub: v1.0.1 → v1.2.0 -- cloud.google.com/go/storage: v1.0.0 → v1.6.0 -- cloud.google.com/go: v0.51.0 → v0.54.0 -- github.com/Azure/go-autorest/autorest/adal: [v0.8.2 → v0.9.5](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.8.2...v0.9.5) -- github.com/Azure/go-autorest/autorest/date: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.2.0...v0.3.0) -- github.com/Azure/go-autorest/autorest/mocks: [v0.3.0 → v0.4.1](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.3.0...v0.4.1) -- github.com/Azure/go-autorest/autorest: [v0.9.6 → v0.11.1](https://github.com/Azure/go-autorest/autorest/compare/v0.9.6...v0.11.1) -- github.com/Azure/go-autorest/logger: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/logger/compare/v0.1.0...v0.2.0) -- github.com/Azure/go-autorest/tracing: [v0.5.0 → v0.6.0](https://github.com/Azure/go-autorest/tracing/compare/v0.5.0...v0.6.0) -- github.com/Microsoft/go-winio: [fc70bd9 → v0.4.15](https://github.com/Microsoft/go-winio/compare/fc70bd9...v0.4.15) -- github.com/aws/aws-sdk-go: [v1.28.2 → v1.35.24](https://github.com/aws/aws-sdk-go/compare/v1.28.2...v1.35.24) -- github.com/blang/semver: [v3.5.0+incompatible → v3.5.1+incompatible](https://github.com/blang/semver/compare/v3.5.0...v3.5.1) -- github.com/checkpoint-restore/go-criu/v4: [v4.0.2 → v4.1.0](https://github.com/checkpoint-restore/go-criu/v4/compare/v4.0.2...v4.1.0) -- github.com/containerd/containerd: [v1.3.3 → v1.4.1](https://github.com/containerd/containerd/compare/v1.3.3...v1.4.1) -- github.com/containerd/ttrpc: [v1.0.0 → v1.0.2](https://github.com/containerd/ttrpc/compare/v1.0.0...v1.0.2) -- github.com/containerd/typeurl: [v1.0.0 → v1.0.1](https://github.com/containerd/typeurl/compare/v1.0.0...v1.0.1) -- github.com/coreos/etcd: [v3.3.10+incompatible → v3.3.13+incompatible](https://github.com/coreos/etcd/compare/v3.3.10...v3.3.13) -- github.com/docker/docker: [aa6a989 → bd33bbf](https://github.com/docker/docker/compare/aa6a989...bd33bbf) -- github.com/go-gl/glfw/v3.3/glfw: [12ad95a → 6f7a984](https://github.com/go-gl/glfw/v3.3/glfw/compare/12ad95a...6f7a984) -- github.com/golang/groupcache: [215e871 → 8c9f03a](https://github.com/golang/groupcache/compare/215e871...8c9f03a) -- github.com/golang/mock: [v1.3.1 → v1.4.1](https://github.com/golang/mock/compare/v1.3.1...v1.4.1) -- github.com/golang/protobuf: [v1.4.2 → v1.4.3](https://github.com/golang/protobuf/compare/v1.4.2...v1.4.3) -- github.com/google/cadvisor: [v0.37.0 → v0.38.5](https://github.com/google/cadvisor/compare/v0.37.0...v0.38.5) -- github.com/google/go-cmp: [v0.4.0 → v0.5.2](https://github.com/google/go-cmp/compare/v0.4.0...v0.5.2) -- github.com/google/pprof: [d4f498a → 1ebb73c](https://github.com/google/pprof/compare/d4f498a...1ebb73c) -- github.com/google/uuid: [v1.1.1 → v1.1.2](https://github.com/google/uuid/compare/v1.1.1...v1.1.2) -- github.com/gorilla/mux: [v1.7.3 → v1.8.0](https://github.com/gorilla/mux/compare/v1.7.3...v1.8.0) -- github.com/gorilla/websocket: [v1.4.0 → v1.4.2](https://github.com/gorilla/websocket/compare/v1.4.0...v1.4.2) -- github.com/jmespath/go-jmespath: [c2b33e8 → v0.4.0](https://github.com/jmespath/go-jmespath/compare/c2b33e8...v0.4.0) -- github.com/karrick/godirwalk: [v1.7.5 → v1.16.1](https://github.com/karrick/godirwalk/compare/v1.7.5...v1.16.1) -- github.com/opencontainers/go-digest: [v1.0.0-rc1 → v1.0.0](https://github.com/opencontainers/go-digest/compare/v1.0.0-rc1...v1.0.0) -- github.com/opencontainers/runc: [819fcc6 → v1.0.0-rc92](https://github.com/opencontainers/runc/compare/819fcc6...v1.0.0-rc92) -- github.com/opencontainers/runtime-spec: [237cc4f → 4d89ac9](https://github.com/opencontainers/runtime-spec/compare/237cc4f...4d89ac9) -- github.com/opencontainers/selinux: [v1.5.2 → v1.6.0](https://github.com/opencontainers/selinux/compare/v1.5.2...v1.6.0) -- github.com/prometheus/procfs: [v0.1.3 → v0.2.0](https://github.com/prometheus/procfs/compare/v0.1.3...v0.2.0) -- github.com/quobyte/api: [v0.1.2 → v0.1.8](https://github.com/quobyte/api/compare/v0.1.2...v0.1.8) -- github.com/spf13/cobra: [v1.0.0 → v1.1.1](https://github.com/spf13/cobra/compare/v1.0.0...v1.1.1) -- github.com/spf13/viper: [v1.4.0 → v1.7.0](https://github.com/spf13/viper/compare/v1.4.0...v1.7.0) -- github.com/storageos/go-api: [343b3ef → v2.2.0+incompatible](https://github.com/storageos/go-api/compare/343b3ef...v2.2.0) -- github.com/stretchr/testify: [v1.4.0 → v1.6.1](https://github.com/stretchr/testify/compare/v1.4.0...v1.6.1) -- github.com/vishvananda/netns: [52d707b → db3c7e5](https://github.com/vishvananda/netns/compare/52d707b...db3c7e5) -- go.etcd.io/etcd: 17cef6e → dd1b699 -- go.opencensus.io: v0.22.2 → v0.22.3 -- golang.org/x/crypto: 75b2880 → 7f63de1 -- golang.org/x/exp: da58074 → 6cc2880 -- golang.org/x/lint: fdd1cda → 738671d -- golang.org/x/net: ab34263 → 69a7880 -- golang.org/x/oauth2: 858c2ad → bf48bf1 -- golang.org/x/sys: ed371f2 → 5cba982 -- golang.org/x/text: v0.3.3 → v0.3.4 -- golang.org/x/time: 555d28b → 3af7569 -- golang.org/x/xerrors: 9bdfabe → 5ec99f8 -- google.golang.org/api: v0.15.1 → v0.20.0 -- google.golang.org/genproto: cb27e3a → 8816d57 -- google.golang.org/grpc: v1.27.0 → v1.27.1 -- google.golang.org/protobuf: v1.24.0 → v1.25.0 -- honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3 -- k8s.io/gengo: 8167cfd → 83324d8 -- k8s.io/klog/v2: v2.2.0 → v2.4.0 -- k8s.io/kube-openapi: 6aeccd4 → d219536 -- k8s.io/system-validators: v1.1.2 → v1.2.0 -- k8s.io/utils: d5654de → 67b214c -- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.14 -- sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → v4.0.2 - -### Removed -- github.com/armon/consul-api: [eb2c6b5](https://github.com/armon/consul-api/tree/eb2c6b5) -- github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) -- github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) -- github.com/xlab/handysort: [fb3537e](https://github.com/xlab/handysort/tree/fb3537e) -- github.com/xordataexchange/crypt: [b2862e3](https://github.com/xordataexchange/crypt/tree/b2862e3) -- vbom.ml/util: db5cfe1 - - - -# v1.20.0-rc.0 - - -## Downloads for v1.20.0-rc.0 - -### Source Code - -filename | sha512 hash --------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes.tar.gz) | acfee8658831f9503fccda0904798405434f17be7064a361a9f34c6ed04f1c0f685e79ca40cef5fcf34e3193bacbf467665e8dc277e0562ebdc929170034b5ae -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-src.tar.gz) | 9d962f8845e1fa221649cf0c0e178f0f03808486c49ea15ab5ec67861ec5aa948cf18bc0ee9b2067643c8332227973dd592e6a4457456a9d9d80e8ef28d5f7c3 - -### Client binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | 062b57f1a450fe01d6184f104d81d376bdf5720010412821e315fd9b1b622a400ac91f996540daa66cee172006f3efade4eccc19265494f1a1d7cc9450f0b50a -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-386.tar.gz) | 86e96d2c2046c5e62e02bef30a6643f25e01f1b3eba256cab7dd61252908540c26cb058490e9cecc5a9bad97d2b577f5968884e9f1a90237e302419f39e068bc -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 619d3afb9ce902368390e71633396010e88e87c5fd848e3adc71571d1d4a25be002588415e5f83afee82460f8a7c9e0bd968335277cb8f8cb51e58d4bb43e64e -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-arm.tar.gz) | 60965150a60ab3d05a248339786e0c7da4b89a04539c3719737b13d71302bac1dd9bcaa427d8a1f84a7b42d0c67801dce2de0005e9e47d21122868b32ac3d40f -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | 688e064f4ef6a17189dbb5af468c279b9de35e215c40500fb97b1d46692d222747023f9e07a7f7ba006400f9532a8912e69d7c5143f956b1dadca144c67ee711 -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | 47b8abc02b42b3b1de67da184921b5801d7e3cb09befac840c85913193fc5ac4e5e3ecfcb57da6b686ff21af9a3bd42ae6949d4744dbe6ad976794340e328b83 -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 971b41d3169f30e6c412e0254c180636abb7ccc8dcee6641b0e9877b69752fc61aa30b76c19c108969df654fe385da3cb3a44dd59d3c28dc45561392d7e08874 -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-windows-386.tar.gz) | 2d34e8387e31531d9aca5655f2f0d18e75b01825dc1c39b7beb73a7b7b610e2ba429e5ca97d5c41a71b67e75e7096c86ab63fda9baab4c0878c1ccb3a1aefac8 -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | f909640f4140693bb871936f10a40e79b43502105d0adb318b35bb7a64a770ad9d05a3a732368ccd3d15d496d75454789165bd1f5c2571da9a00569b3e6c007c - -### Server binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | 0ea4458ae34108c633b4d48f1f128c6274dbc82b613492e78b3e0a2f656ac0df0bb9a75124e15d67c8e81850adcecf19f4ab0234c17247ee7ddf84f2df3e5eaa -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-arm.tar.gz) | aef6a4d457faa29936603370f29a8523bb274211c3cb5101bd31aaf469c91ba6bd149ea99a4ccdd83352cf37e4d6508c5ee475ec10292bccd2f77ceea31e1c28 -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | 4829f473e9d60f9929ad17c70fdc2b6b6509ed75418be0b23a75b28580949736cb5b0bd6382070f93aa0a2a8863f0b1596daf965186ca749996c29d03ef7d8b8 -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | 9ab0790d382a3e28df1c013762c09da0085449cfd09d176d80be932806c24a715ea829be0075c3e221a2ad9cf06e726b4c39ab41987c1fb0fee2563e48206763 -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 98670b587e299856dd9821b7517a35f9a65835b915b153de08b66c54d82160438b66f774bf5306c07bc956d70ff709860bc23162225da5e89f995d3fdc1f0122 - -### Node binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | 699e9c8d1837198312eade8eb6fec390f6a2fea9e08207d2f58e8bb6e3e799028aca69e4670aac0a4ba7cf0af683aee2c158bf78cc520c80edc876c8d94d521a -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-arm.tar.gz) | f3b5eab0669490e3cd7e802693daf3555d08323dfff6e73a881fce00fed4690e8bdaf1610278d9de74036ca37631016075e5695a02158b7d3e7582b20ef7fa35 -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | e5012f77363561a609aaf791baaa17d09009819c4085a57132e5feb5366275a54640094e6ed1cba527f42b586c6d62999c2a5435edf5665ff0e114db4423c2ae -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 2a6d6501620b1a9838dff05c66a40260cc22154a28027813346eb16e18c386bc3865298a46a0f08da71cd55149c5e7d07c4c4c431b4fd231486dd9d716548adb -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | 5eca02777519e31428a1e5842fe540b813fb8c929c341bbc71dcfd60d98deb89060f8f37352e8977020e21e053379eead6478eb2d54ced66fb9d38d5f3142bf0 -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | 8ace02e7623dff894e863a2e0fa7dfb916368431d1723170713fe82e334c0ae0481b370855b71e2561de0fb64fed124281be604761ec08607230b66fb9ed1c03 - -## Changelog since v1.20.0-beta.2 - -## Changes by Kind - -### Feature - -- Kubernetes is now built using go1.15.5 - - build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) ([#95776](https://github.com/kubernetes/kubernetes/pull/95776), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] +- Azure file migration graduates to beta, with CSIMigrationAzureFile flag off by default + as it requires installation of AzureFile CSI Driver. Users should enable CSIMigration and + CSIMigrationAzureFile features and install the [AzureFile CSI Driver](https://github.com/kubernetes-sigs/azurefile-csi-driver) + to avoid disruption to existing Pod and PVC objects at that time. Azure File CSI driver does not support using same persistent + volume with different fsgroups. When CSI migration is enabled for azurefile driver, such case is not supported. + (there is a case we support where volume is mounted with 0777 and then it readable/writable by everyone) ([#96293](https://github.com/kubernetes/kubernetes/pull/96293), [@andyzhangx](https://github.com/andyzhangx)) +- Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97935](https://github.com/kubernetes/kubernetes/pull/97935), [@adeniyistephen](https://github.com/adeniyistephen)) [SIG Release and Testing] +- Set kubelet option `--volume-stats-agg-period` to negative value to disable volume calculations. ([#96675](https://github.com/kubernetes/kubernetes/pull/96675), [@pacoxu](https://github.com/pacoxu)) [SIG Node] ### Failing Test -- Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources ([#96742](https://github.com/kubernetes/kubernetes/pull/96742), [@spencerhance](https://github.com/spencerhance)) [SIG Network and Testing] -- The Conformance test "validates that there is no conflict between pods with same hostPort but different hostIP and protocol" now validates the connectivity to each hostPort, in addition to the functionality. ([#96627](https://github.com/kubernetes/kubernetes/pull/96627), [@aojea](https://github.com/aojea)) [SIG Scheduling and Testing] +- Escape the special characters like `[`, `]` and ` ` that exist in vsphere windows path ([#98830](https://github.com/kubernetes/kubernetes/pull/98830), [@liyanhui1228](https://github.com/liyanhui1228)) [SIG Storage and Windows] +- Kube-proxy: fix a bug on UDP `NodePort` Services where stale connection tracking entries may blackhole the traffic directed to the `NodePort` ([#98305](https://github.com/kubernetes/kubernetes/pull/98305), [@aojea](https://github.com/aojea)) +- Kubelet: fixes a bug in the HostPort dockershim implementation that caused the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. ([#98755](https://github.com/kubernetes/kubernetes/pull/98755), [@aojea](https://github.com/aojea)) [SIG Cloud Provider, Network and Node] ### Bug or Regression -- Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ ([#96716](https://github.com/kubernetes/kubernetes/pull/96716), [@tosi3k](https://github.com/tosi3k)) [SIG Cloud Provider, Scalability and Testing] -- Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. ([#96727](https://github.com/kubernetes/kubernetes/pull/96727), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] -- Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. ([#96185](https://github.com/kubernetes/kubernetes/pull/96185), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery and Testing] -- Fix: resize Azure disk issue when it's in attached state ([#96705](https://github.com/kubernetes/kubernetes/pull/96705), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. ([#96421](https://github.com/kubernetes/kubernetes/pull/96421), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG API Machinery and Instrumentation] -- Fixes code generation for non-namespaced create subresources fake client test. ([#96586](https://github.com/kubernetes/kubernetes/pull/96586), [@Doude](https://github.com/Doude)) [SIG API Machinery] -- HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. ([#95981](https://github.com/kubernetes/kubernetes/pull/95981), [@caesarxuchao](https://github.com/caesarxuchao)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node] -- Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade ([#96907](https://github.com/kubernetes/kubernetes/pull/96907), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] -- Reduce volume name length for vsphere volumes ([#96533](https://github.com/kubernetes/kubernetes/pull/96533), [@gnufied](https://github.com/gnufied)) [SIG Storage] -- Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. ([#96810](https://github.com/kubernetes/kubernetes/pull/96810), [@liggitt](https://github.com/liggitt)) [SIG Node] - -## Dependencies - -### Added -_Nothing has changed._ - -### Changed -- github.com/google/cadvisor: [v0.38.4 → v0.38.5](https://github.com/google/cadvisor/compare/v0.38.4...v0.38.5) - -### Removed -_Nothing has changed._ - - - -# v1.20.0-beta.2 - - -## Downloads for v1.20.0-beta.2 - -### Source Code - -filename | sha512 hash --------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes.tar.gz) | fe769280aa623802a949b6a35fbddadbba1d6f9933a54132a35625683719595ecf58096a9aa0f7456f8d4931774df21bfa98e148bc3d85913f1da915134f77bd -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-src.tar.gz) | ce1c8d97c52e5189af335d673bd7e99c564816f6adebf249838f7e3f0e920f323b4e398a5d163ea767091497012ec38843c59ff14e6fdd07683b682135eed645 - -### Client binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-darwin-amd64.tar.gz) | d6c14bd0f6702f4bbdf14a6abdfa4e5936de5b4efee38aa86c2bd7272967ec6d7868b88fc00ad4a7c3a20717a35e6be2b84e56dec04154fd702315f641409f7c -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-linux-386.tar.gz) | b923c44cb0acb91a8f6fd442c2168aa6166c848f5d037ce50a7cb11502be3698db65836b373c916f75b648d6ac8d9158807a050eecc4e1c77cffa25b386c8cdb -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-linux-amd64.tar.gz) | 8cae14146a9034dcd4e9d69d5d700f195a77aac35f629a148960ae028ed8b4fe12213993fe3e6e464b4b3e111adebe6f3dd7ca0accc70c738ed5cfd8993edd7c -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-linux-arm.tar.gz) | 1f54e5262a0432945ead57fcb924e6bfedd9ea76db1dd9ebd946787a2923c247cf16e10505307b47e365905a1b398678dac5af0f433c439c158a33e08362d97b -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-linux-arm64.tar.gz) | 31cf79c01e4878a231b4881fe3ed5ef790bd5fb5419388438d3f8c6a2129e655aba9e00b8e1d77e0bc5d05ecc75cf4ae02cf8266788822d0306c49c85ee584ed -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-linux-ppc64le.tar.gz) | 2527948c40be2e16724d939316ad5363f15aa22ebf42d59359d8b6f757d30cfef6447434cc93bc5caa5a23a6a00a2da8d8191b6441e06bba469d9d4375989a97 -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-linux-s390x.tar.gz) | b777ad764b3a46651ecb0846e5b7f860bb2c1c4bd4d0fcc468c6ccffb7d3b8dcb6dcdd73b13c16ded7219f91bba9f1e92f9258527fd3bb162b54d7901ac303ff -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-windows-386.tar.gz) | 8a2f58aaab01be9fe298e4d01456536047cbdd39a37d3e325c1f69ceab3a0504998be41a9f41a894735dfc4ed22bed02591eea5f3c75ce12d9e95ba134e72ec5 -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-client-windows-amd64.tar.gz) | 2f69cda177a178df149f5de66b7dba7f5ce14c1ffeb7c8d7dc4130c701b47d89bb2fbe74e7a262f573e4d21dee2c92414d050d7829e7c6fc3637a9d6b0b9c5c1 - -### Server binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-server-linux-amd64.tar.gz) | 3ecaac0213d369eab691ac55376821a80df5013cb12e1263f18d1c236a9e49d42b3cea422175556d8f929cdf3109b22c0b6212ac0f2e80cc7a5f4afa3aba5f24 -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-server-linux-arm.tar.gz) | 580030b57ff207e177208fec0801a43389cae10cc2c9306327d354e7be6a055390184531d54b6742e0983550b7a76693cc4a705c2d2f4ac30495cf63cef26b9b -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-server-linux-arm64.tar.gz) | 3e3286bd54671549fbef0dfdaaf1da99bc5c3efb32cc8d1e1985d9926520cea0c43bcf7cbcbbc8b1c1a95eab961255693008af3bb1ba743362998b5f0017d6d7 -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-server-linux-ppc64le.tar.gz) | 9fa051e7e97648e97e26b09ab6d26be247b41b1a5938d2189204c9e6688e455afe76612bbcdd994ed5692935d0d960bd96dc222bce4b83f61d62557752b9d75b -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-server-linux-s390x.tar.gz) | fa85d432eff586f30975c95664ac130b9f5ae02dc52b97613ed7a41324496631ea11d1a267daba564cf2485a9e49707814d86bbd3175486c7efc8b58a9314af5 - -### Node binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-node-linux-amd64.tar.gz) | 86e631f95fe670b467ead2b88d34e0364eaa275935af433d27cc378d82dcaa22041ccce40f5fa9561b9656dadaa578dc018ad458a59b1690d35f86dca4776b5c -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-node-linux-arm.tar.gz) | a8754ff58a0e902397056b8615ab49af07aca347ba7cc4a812c238e3812234862270f25106b6a94753b157bb153b8eae8b39a01ed67384774d798598c243583b -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-node-linux-arm64.tar.gz) | 28d727d7d08e2c856c9b4a574ef2dbf9e37236a0555f7ec5258b4284fa0582fb94b06783aaf50bf661f7503d101fbd70808aba6de02a2f0af94db7d065d25947 -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-node-linux-ppc64le.tar.gz) | a1283449f1a0b155c11449275e9371add544d0bdd4609d6dc737ed5f7dd228e84e24ff249613a2a153691627368dd894ad64f4e6c0010eecc6efd2c13d4fb133 -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-node-linux-s390x.tar.gz) | 5806028ba15a6a9c54a34f90117bc3181428dbb0e7ced30874c9f4a953ea5a0e9b2c73e6b1e2545e1b4e5253e9c7691588538b44cdfa666ce6865964b92d2fa8 -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.2/kubernetes-node-windows-amd64.tar.gz) | d5327e3b7916c78777b9b69ba0f3758c3a8645c67af80114a0ae52babd7af27bb504febbaf51b1bfe5bd2d74c8c5c573471e1cb449f2429453f4b1be9d5e682a - -## Changelog since v1.20.0-beta.1 - -## Urgent Upgrade Notes - -### (No, really, you MUST read this before you upgrade) - - - A bug was fixed in kubelet where exec probe timeouts were not respected. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts. - - This change in behavior may be unexpected for some clusters and can be disabled by turning off the ExecProbeTimeout feature gate. This gate will be locked and removed in future releases so that exec probe timeouts are always respected. ([#94115](https://github.com/kubernetes/kubernetes/pull/94115), [@andrewsykim](https://github.com/andrewsykim)) [SIG Node and Testing] - - For CSI drivers, kubelet no longer creates the target_path for NodePublishVolume in accordance with the CSI spec. Kubelet also no longer checks if staging and target paths are mounts or corrupted. CSI drivers need to be idempotent and do any necessary mount verification. ([#88759](https://github.com/kubernetes/kubernetes/pull/88759), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] - - Kubeadm: - - The label applied to control-plane nodes "node-role.kubernetes.io/master" is now deprecated and will be removed in a future release after a GA deprecation period. - - Introduce a new label "node-role.kubernetes.io/control-plane" that will be applied in parallel to "node-role.kubernetes.io/master" until the removal of the "node-role.kubernetes.io/master" label. - - Make "kubeadm upgrade apply" add the "node-role.kubernetes.io/control-plane" label on existing nodes that only have the "node-role.kubernetes.io/master" label during upgrade. - - Please adapt your tooling built on top of kubeadm to use the "node-role.kubernetes.io/control-plane" label. - - - The taint applied to control-plane nodes "node-role.kubernetes.io/master:NoSchedule" is now deprecated and will be removed in a future release after a GA deprecation period. - - Apply toleration for a new, future taint "node-role.kubernetes.io/control-plane:NoSchedule" to the kubeadm CoreDNS / kube-dns managed manifests. Note that this taint is not yet applied to kubeadm control-plane nodes. - - Please adapt your workloads to tolerate the same future taint preemptively. - - For more details see: http://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md ([#95382](https://github.com/kubernetes/kubernetes/pull/95382), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - -## Changes by Kind - -### Deprecation - -- Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. ([#94624](https://github.com/kubernetes/kubernetes/pull/94624), [@dims](https://github.com/dims)) [SIG Node] -- Kubectl: deprecate --delete-local-data ([#95076](https://github.com/kubernetes/kubernetes/pull/95076), [@dougsland](https://github.com/dougsland)) [SIG CLI, Cloud Provider and Scalability] - -### API Change - -- API priority and fairness graduated to beta - 1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. ([#96527](https://github.com/kubernetes/kubernetes/pull/96527), [@adtac](https://github.com/adtac)) [SIG API Machinery and Testing] -- Add LoadBalancerIPMode feature gate ([#92312](https://github.com/kubernetes/kubernetes/pull/92312), [@Sh4d1](https://github.com/Sh4d1)) [SIG Apps, CLI, Cloud Provider and Network] -- Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest ([#95741](https://github.com/kubernetes/kubernetes/pull/95741), [@katiewasnothere](https://github.com/katiewasnothere)) [SIG Node] -- Add a 'serving' and `terminating` condition to the EndpointSlice API. - - `serving` tracks the readiness of endpoints regardless of their terminating state. This is distinct from `ready` since `ready` is only true when pods are not terminating. - `terminating` is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. ([#92968](https://github.com/kubernetes/kubernetes/pull/92968), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps and Network] -- Add support for hugepages to downward API ([#86102](https://github.com/kubernetes/kubernetes/pull/86102), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] -- Adds kubelet alpha feature, `GracefulNodeShutdown` which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. ([#96129](https://github.com/kubernetes/kubernetes/pull/96129), [@bobbypage](https://github.com/bobbypage)) [SIG Node] -- AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. ([#96327](https://github.com/kubernetes/kubernetes/pull/96327), [@robscott](https://github.com/robscott)) [SIG Apps and Network] -- Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter - Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. ([#92744](https://github.com/kubernetes/kubernetes/pull/92744), [@uablrek](https://github.com/uablrek)) [SIG Apps and Network] -- Document that ServiceTopology feature is required to use `service.spec.topologyKeys`. ([#96528](https://github.com/kubernetes/kubernetes/pull/96528), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] -- EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate. - - EndpointSlice topology field will be deprecated in an upcoming release. - - EndpointSlice "IP" address type is formally removed after being deprecated in Kubernetes 1.17. - - The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. ([#96440](https://github.com/kubernetes/kubernetes/pull/96440), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps and Network] -- Fewer candidates are enumerated for preemption to improve performance in large clusters ([#94814](https://github.com/kubernetes/kubernetes/pull/94814), [@adtac](https://github.com/adtac)) [SIG Scheduling] -- If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric `serviceaccount_stale_tokens_total` to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting `kube-apiserver` with flag `--service-account-extend-token-expiration=false` ([#96273](https://github.com/kubernetes/kubernetes/pull/96273), [@zshihang](https://github.com/zshihang)) [SIG API Machinery and Auth] -- Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. ([#94196](https://github.com/kubernetes/kubernetes/pull/94196), [@andrewsykim](https://github.com/andrewsykim)) [SIG Node and Release] -- Kube-apiserver now deletes expired kube-apiserver Lease objects: - - The feature is under feature gate `APIServerIdentity`. - - A flag is added to kube-apiserver: `identity-lease-garbage-collection-check-period-seconds` ([#95895](https://github.com/kubernetes/kubernetes/pull/95895), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery, Apps, Auth and Testing] -- Move configurable fsgroup change policy for pods to beta ([#96376](https://github.com/kubernetes/kubernetes/pull/96376), [@gnufied](https://github.com/gnufied)) [SIG Apps and Storage] -- New flag is introduced, i.e. --topology-manager-scope=container|pod. - The default value is the "container" scope. ([#92967](https://github.com/kubernetes/kubernetes/pull/92967), [@cezaryzukowski](https://github.com/cezaryzukowski)) [SIG Instrumentation, Node and Testing] -- NodeAffinity plugin can be configured with AddedAffinity. ([#96202](https://github.com/kubernetes/kubernetes/pull/96202), [@alculquicondor](https://github.com/alculquicondor)) [SIG Node, Scheduling and Testing] -- Promote RuntimeClass feature to GA. - Promote node.k8s.io API groups from v1beta1 to v1. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] -- Reminder: The labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" are deprecated in favor of "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" respectively. All users of the "failure-domain.beta..." labels should switch to the "topology..." equivalents. ([#96033](https://github.com/kubernetes/kubernetes/pull/96033), [@thockin](https://github.com/thockin)) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing] -- The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBSVC is enabled. - "action required" - The feature gate is disabled by default. The user has to enable it for the API Server. ([#94028](https://github.com/kubernetes/kubernetes/pull/94028), [@janosi](https://github.com/janosi)) [SIG API Machinery and Apps] -- This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in `CSIDriverSpec`. ([#93130](https://github.com/kubernetes/kubernetes/pull/93130), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing] -- Users can try the cronjob controller v2 using the feature gate. This will be the default controller in future releases. ([#93370](https://github.com/kubernetes/kubernetes/pull/93370), [@alaypatel07](https://github.com/alaypatel07)) [SIG API Machinery, Apps, Auth and Testing] -- VolumeSnapshotDataSource moves to GA in 1.20 release ([#95282](https://github.com/kubernetes/kubernetes/pull/95282), [@xing-yang](https://github.com/xing-yang)) [SIG Apps] - -### Feature - -- **Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.**: ([#95896](https://github.com/kubernetes/kubernetes/pull/95896), [@zshihang](https://github.com/zshihang)) [SIG API Machinery and Cluster Lifecycle] -- A new set of alpha metrics are reported by the Kubernetes scheduler under the `/metrics/resources` endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. ([#94866](https://github.com/kubernetes/kubernetes/pull/94866), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Instrumentation, Node and Scheduling] -- Add --experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs ([#96370](https://github.com/kubernetes/kubernetes/pull/96370), [@serathius](https://github.com/serathius)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] -- Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. - This feature allows the storage migrator to manage storage migration for built-in resources. - Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. ([#93873](https://github.com/kubernetes/kubernetes/pull/93873), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery, Auth and Testing] -- Add a new `vSphere` metric: `cloudprovider_vsphere_vcenter_versions`. It's content show `vCenter` hostnames with the associated server version. ([#94526](https://github.com/kubernetes/kubernetes/pull/94526), [@Danil-Grigorev](https://github.com/Danil-Grigorev)) [SIG Cloud Provider and Instrumentation] -- Add feature to size memory backed volumes ([#94444](https://github.com/kubernetes/kubernetes/pull/94444), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Storage and Testing] -- Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. ([#92466](https://github.com/kubernetes/kubernetes/pull/92466), [@mborsz](https://github.com/mborsz)) [SIG API Machinery, Auth and Instrumentation] -- Add pod_ based CPU and memory metrics to Kubelet's /metrics/resource endpoint ([#95839](https://github.com/kubernetes/kubernetes/pull/95839), [@egernst](https://github.com/egernst)) [SIG Instrumentation, Node and Testing] -- Adds a headless service on node-local-cache addon. ([#88412](https://github.com/kubernetes/kubernetes/pull/88412), [@stafot](https://github.com/stafot)) [SIG Cloud Provider and Network] -- CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in list will continue being preserved, and fail validation if not nullable. ([#95423](https://github.com/kubernetes/kubernetes/pull/95423), [@apelisse](https://github.com/apelisse)) [SIG API Machinery] -- E2e test for PodFsGroupChangePolicy ([#96247](https://github.com/kubernetes/kubernetes/pull/96247), [@saikat-royc](https://github.com/saikat-royc)) [SIG Storage and Testing] -- Gradudate the Pod Resources API to G.A - Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API ([#92165](https://github.com/kubernetes/kubernetes/pull/92165), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Instrumentation, Node and Testing] -- Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. ([#95603](https://github.com/kubernetes/kubernetes/pull/95603), [@soltysh](https://github.com/soltysh)) [SIG API Machinery] -- Kube-apiserver now maintains a Lease object to identify itself: - - The feature is under feature gate `APIServerIdentity`. - - Two flags are added to kube-apiserver: `identity-lease-duration-seconds`, `identity-lease-renew-interval-seconds` ([#95533](https://github.com/kubernetes/kubernetes/pull/95533), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] -- Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with `--etcd-healthcheck-timeout`. The default timeout is 2 seconds, matching the previous behavior. ([#93244](https://github.com/kubernetes/kubernetes/pull/93244), [@Sh4d1](https://github.com/Sh4d1)) [SIG API Machinery] -- Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. ([#95292](https://github.com/kubernetes/kubernetes/pull/95292), [@dougsland](https://github.com/dougsland)) [SIG CLI] -- Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. ([#96071](https://github.com/kubernetes/kubernetes/pull/96071), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] -- Support custom tags for cloud provider managed resources ([#96450](https://github.com/kubernetes/kubernetes/pull/96450), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Support customize load balancer health probe protocol and request path ([#96338](https://github.com/kubernetes/kubernetes/pull/96338), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Support multiple standard load balancers in one cluster ([#96111](https://github.com/kubernetes/kubernetes/pull/96111), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- The beta `RootCAConfigMap` feature gate is enabled by default and causes kube-controller-manager to publish a "kube-root-ca.crt" ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. ([#96197](https://github.com/kubernetes/kubernetes/pull/96197), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Apps, Auth and Testing] -- The kubelet_runtime_operations_duration_seconds metric got additional buckets of 60, 300, 600, 900 and 1200 seconds ([#96054](https://github.com/kubernetes/kubernetes/pull/96054), [@alvaroaleman](https://github.com/alvaroaleman)) [SIG Instrumentation and Node] -- There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. ([#95719](https://github.com/kubernetes/kubernetes/pull/95719), [@tsmetana](https://github.com/tsmetana)) [SIG Apps, Instrumentation, Storage and Testing] -- Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer ([#95863](https://github.com/kubernetes/kubernetes/pull/95863), [@RaunakShah](https://github.com/RaunakShah)) [SIG Cloud Provider, Storage and Testing] -- Warns user when executing kubectl apply/diff to resource currently being deleted. ([#95544](https://github.com/kubernetes/kubernetes/pull/95544), [@SaiHarshaK](https://github.com/SaiHarshaK)) [SIG CLI] -- `kubectl alpha debug` has graduated to beta and is now `kubectl debug`. ([#96138](https://github.com/kubernetes/kubernetes/pull/96138), [@verb](https://github.com/verb)) [SIG CLI and Testing] -- `kubectl debug` gains support for changing container images when copying a pod for debugging, similar to how `kubectl set image` works. See `kubectl help debug` for more information. ([#96058](https://github.com/kubernetes/kubernetes/pull/96058), [@verb](https://github.com/verb)) [SIG CLI] - -### Documentation - -- Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers: - - removes experimental warning for InstancesV2 - - document that implementation of InstancesV2 will disable calls to Zones - - deprecate Zones in favor of InstancesV2 ([#96397](https://github.com/kubernetes/kubernetes/pull/96397), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] - -### Bug or Regression - -- Change plugin name in fsgroupapplymetrics of csi and flexvolume to distinguish different driver ([#95892](https://github.com/kubernetes/kubernetes/pull/95892), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation, Storage and Testing] -- Clear UDP conntrack entry on endpoint changes when using nodeport ([#71573](https://github.com/kubernetes/kubernetes/pull/71573), [@JacobTanenbaum](https://github.com/JacobTanenbaum)) [SIG Network] -- Exposes and sets a default timeout for the TokenReview client for DelegatingAuthenticationOptions ([#96217](https://github.com/kubernetes/kubernetes/pull/96217), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Cloud Provider] -- Fix CVE-2020-8555 for Quobyte client connections. ([#95206](https://github.com/kubernetes/kubernetes/pull/95206), [@misterikkit](https://github.com/misterikkit)) [SIG Storage] -- Fix IP fragmentation of UDP and TCP packets not supported issues on LoadBalancer rules ([#96464](https://github.com/kubernetes/kubernetes/pull/96464), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Fix a bug that DefaultPreemption plugin is disabled when using (legacy) scheduler policy. ([#96439](https://github.com/kubernetes/kubernetes/pull/96439), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] -- Fix bug in JSON path parser where an error occurs when a range is empty ([#95933](https://github.com/kubernetes/kubernetes/pull/95933), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] -- Fix client-go prometheus metrics to correctly present the API path accessed in some environments. ([#74363](https://github.com/kubernetes/kubernetes/pull/74363), [@aanm](https://github.com/aanm)) [SIG API Machinery] -- Fix memory leak in kube-apiserver when underlying time goes forth and back. ([#96266](https://github.com/kubernetes/kubernetes/pull/96266), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery] -- Fix paging issues when Azure API returns empty values with non-empty nextLink ([#96211](https://github.com/kubernetes/kubernetes/pull/96211), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] -- Fix pull image error from multiple ACRs using azure managed identity ([#96355](https://github.com/kubernetes/kubernetes/pull/96355), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix vSphere volumes that could be erroneously attached to wrong node ([#96224](https://github.com/kubernetes/kubernetes/pull/96224), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] -- Fixed a bug that prevents kubectl to validate CRDs with schema using x-kubernetes-preserve-unknown-fields on object fields. ([#96369](https://github.com/kubernetes/kubernetes/pull/96369), [@gautierdelorme](https://github.com/gautierdelorme)) [SIG API Machinery and Testing] -- For vSphere Cloud Provider, If VM of worker node is deleted, the node will also be deleted by node controller ([#92608](https://github.com/kubernetes/kubernetes/pull/92608), [@lubronzhan](https://github.com/lubronzhan)) [SIG Cloud Provider] -- HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. ([#95981](https://github.com/kubernetes/kubernetes/pull/95981), [@caesarxuchao](https://github.com/caesarxuchao)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node] -- If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400. - - If the user specifies a timeout in the request URL that exceeds the maximum request deadline allowed by the apiserver, the request will be aborted with an HTTP 400. ([#96061](https://github.com/kubernetes/kubernetes/pull/96061), [@tkashem](https://github.com/tkashem)) [SIG API Machinery, Network and Testing] -- Improve error messages related to nodePort endpoint changes conntrack entries cleanup. ([#96251](https://github.com/kubernetes/kubernetes/pull/96251), [@ravens](https://github.com/ravens)) [SIG Network] -- Print go stack traces at -v=4 and not -v=2 ([#94663](https://github.com/kubernetes/kubernetes/pull/94663), [@soltysh](https://github.com/soltysh)) [SIG CLI] -- Remove ready file and its directory (which is created during volume SetUp) during emptyDir volume TearDown. ([#95770](https://github.com/kubernetes/kubernetes/pull/95770), [@jingxu97](https://github.com/jingxu97)) [SIG Storage] -- Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of `OwnerRefInvalidNamespace` are recorded when namespace mismatches between child and owner objects are detected. - - A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted. - - A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. ([#92743](https://github.com/kubernetes/kubernetes/pull/92743), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Testing] -- Skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:162]: Driver azure-disk doesn't support snapshot type DynamicSnapshot -- skipping - skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:185]: Driver azure-disk doesn't support ntfs -- skipping ([#96144](https://github.com/kubernetes/kubernetes/pull/96144), [@qinpingli](https://github.com/qinpingli)) [SIG Storage and Testing] -- The AWS network load balancer attributes can now be specified during service creation ([#95247](https://github.com/kubernetes/kubernetes/pull/95247), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] -- The kube-apiserver will no longer serve APIs that should have been deleted in GA non-alpha levels. Alpha levels will continue to serve the removed APIs so that CI doesn't immediately break. ([#96525](https://github.com/kubernetes/kubernetes/pull/96525), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] -- Update max azure data disk count map ([#96308](https://github.com/kubernetes/kubernetes/pull/96308), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Update the route table tag in the route reconcile loop ([#96545](https://github.com/kubernetes/kubernetes/pull/96545), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Volume binding: report UnschedulableAndUnresolvable status instead of an error when bound PVs not found ([#95541](https://github.com/kubernetes/kubernetes/pull/95541), [@cofyc](https://github.com/cofyc)) [SIG Apps, Scheduling and Storage] -- [kubectl] Fail when local source file doesn't exist ([#90333](https://github.com/kubernetes/kubernetes/pull/90333), [@bamarni](https://github.com/bamarni)) [SIG CLI] +- AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. ([#96873](https://github.com/kubernetes/kubernetes/pull/96873), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] +- All data is no longer automatically deleted when a failure is detected during creation of the volume data file on a CSI volume. Now only the data file and volume path is removed. ([#96021](https://github.com/kubernetes/kubernetes/pull/96021), [@huffmanca](https://github.com/huffmanca)) +- Clean ReplicaSet by revision instead of creation timestamp in deployment controller ([#97407](https://github.com/kubernetes/kubernetes/pull/97407), [@waynepeking348](https://github.com/waynepeking348)) [SIG Apps] +- Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. ([#98133](https://github.com/kubernetes/kubernetes/pull/98133), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Client-go exec credential plugins will pass stdin only when interactive terminal is detected on stdin. This fixes a bug where previously it was checking if **stdout** is an interactive terminal. ([#99654](https://github.com/kubernetes/kubernetes/pull/99654), [@ankeesler](https://github.com/ankeesler)) +- Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs ([#97029](https://github.com/kubernetes/kubernetes/pull/97029), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider and Testing] +- Cluster Autoscaler version bump to v1.20.0 ([#97011](https://github.com/kubernetes/kubernetes/pull/97011), [@towca](https://github.com/towca)) +- Creating a PVC with DataSource should fail for non-CSI plugins. ([#97086](https://github.com/kubernetes/kubernetes/pull/97086), [@xing-yang](https://github.com/xing-yang)) [SIG Apps and Storage] +- EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99345](https://github.com/kubernetes/kubernetes/pull/99345), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- EndpointSlice controllers are less likely to create duplicate EndpointSlices. ([#100103](https://github.com/kubernetes/kubernetes/pull/100103), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99756](https://github.com/kubernetes/kubernetes/pull/99756), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- Ensure all vSphere nodes are are tracked by volume attach-detach controller ([#96689](https://github.com/kubernetes/kubernetes/pull/96689), [@gnufied](https://github.com/gnufied)) +- Ensure empty string annotations are copied over in rollbacks. ([#94858](https://github.com/kubernetes/kubernetes/pull/94858), [@waynepeking348](https://github.com/waynepeking348)) +- Ensure only one LoadBalancer rule is created when HA mode is enabled ([#99825](https://github.com/kubernetes/kubernetes/pull/99825), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] +- Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. ([#95664](https://github.com/kubernetes/kubernetes/pull/95664), [@DirectXMan12](https://github.com/DirectXMan12)) [SIG API Machinery] +- Explicitly pass `KUBE_BUILD_CONFORMANCE=y` in `package-tarballs` to reenable building the conformance tarballs. ([#100571](https://github.com/kubernetes/kubernetes/pull/100571), [@puerco](https://github.com/puerco)) +- Fix Azure file migration e2e test failure when CSIMigration is turned on. ([#97877](https://github.com/kubernetes/kubernetes/pull/97877), [@andyzhangx](https://github.com/andyzhangx)) +- Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// ([#96821](https://github.com/kubernetes/kubernetes/pull/96821), [@wongma7](https://github.com/wongma7)) [SIG Storage] +- Fix CVE-2020-8555 for Gluster client connections. ([#97922](https://github.com/kubernetes/kubernetes/pull/97922), [@liggitt](https://github.com/liggitt)) [SIG Storage] +- Fix NPE in ephemeral storage eviction ([#98261](https://github.com/kubernetes/kubernetes/pull/98261), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Fix PermissionDenied issue on SMB mount for Windows ([#99550](https://github.com/kubernetes/kubernetes/pull/99550), [@andyzhangx](https://github.com/andyzhangx)) +- Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid ([#99514](https://github.com/kubernetes/kubernetes/pull/99514), [@mikkeloscar](https://github.com/mikkeloscar)) [SIG Apps and Autoscaling] +- Fix cgroup handling for systemd with cgroup v2 ([#98365](https://github.com/kubernetes/kubernetes/pull/98365), [@odinuge](https://github.com/odinuge)) [SIG Node] +- Fix counting error in service/nodeport/loadbalancer quota check ([#97451](https://github.com/kubernetes/kubernetes/pull/97451), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Network and Testing] +- Fix errors when accessing Windows container stats for Dockershim ([#98510](https://github.com/kubernetes/kubernetes/pull/98510), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node and Windows] +- Fix kube-proxy container image architecture for non amd64 images. ([#98526](https://github.com/kubernetes/kubernetes/pull/98526), [@saschagrunert](https://github.com/saschagrunert)) +- Fix missing cadvisor machine metrics. ([#97006](https://github.com/kubernetes/kubernetes/pull/97006), [@lingsamuel](https://github.com/lingsamuel)) [SIG Node] +- Fix nil VMSS name when setting service to auto mode ([#97366](https://github.com/kubernetes/kubernetes/pull/97366), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Fix privileged config of Pod Sandbox which was previously ignored. ([#96877](https://github.com/kubernetes/kubernetes/pull/96877), [@xeniumlee](https://github.com/xeniumlee)) +- Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable ([#95269](https://github.com/kubernetes/kubernetes/pull/95269), [@SataQiu](https://github.com/SataQiu)) [SIG Node] +- Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. ([#97980](https://github.com/kubernetes/kubernetes/pull/97980), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node] +- Fix to recover CSI volumes from certain dangling attachments ([#96617](https://github.com/kubernetes/kubernetes/pull/96617), [@yuga711](https://github.com/yuga711)) [SIG Apps and Storage] +- Fix: azure file latency issue for metadata-heavy workloads ([#97082](https://github.com/kubernetes/kubernetes/pull/97082), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] +- Fixed Cinder volume IDs on OpenStack Train ([#96673](https://github.com/kubernetes/kubernetes/pull/96673), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider] +- Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. ([#97013](https://github.com/kubernetes/kubernetes/pull/97013), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] +- Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. ([#97174](https://github.com/kubernetes/kubernetes/pull/97174), [@hanlins](https://github.com/hanlins)) [SIG Node] +- Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) ([#99613](https://github.com/kubernetes/kubernetes/pull/99613), [@xh4n3](https://github.com/xh4n3)) [SIG Network] +- Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. + Added iptables rules to allow healthcheck nodeport traffic. ([#97824](https://github.com/kubernetes/kubernetes/pull/97824), [@hanlins](https://github.com/hanlins)) [SIG Network] +- Fixed a bug that the kubelet cannot start on BtrfS. ([#98042](https://github.com/kubernetes/kubernetes/pull/98042), [@gjkim42](https://github.com/gjkim42)) [SIG Node] +- Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. ([#95783](https://github.com/kubernetes/kubernetes/pull/95783), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] +- Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children ([#98068](https://github.com/kubernetes/kubernetes/pull/98068), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps] +- Fixed authentication_duration_seconds metric scope. Previously, it included whole apiserver request duration which yields inaccurate results. ([#99944](https://github.com/kubernetes/kubernetes/pull/99944), [@marseel](https://github.com/marseel)) +- Fixed bug in CPUManager with race on container map access ([#97427](https://github.com/kubernetes/kubernetes/pull/97427), [@klueska](https://github.com/klueska)) [SIG Node] +- Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. ([#99315](https://github.com/kubernetes/kubernetes/pull/99315), [@iwankgb](https://github.com/iwankgb)) [SIG Node] +- Fixed cleanup of block devices when /var/lib/kubelet is a symlink. ([#96889](https://github.com/kubernetes/kubernetes/pull/96889), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] +- Fixed no effect namespace when exposing deployment with --dry-run=client. ([#97492](https://github.com/kubernetes/kubernetes/pull/97492), [@masap](https://github.com/masap)) [SIG CLI] +- Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. ([#98311](https://github.com/kubernetes/kubernetes/pull/98311), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] +- Fixes a bug of identifying the correct containerd process. ([#97888](https://github.com/kubernetes/kubernetes/pull/97888), [@pacoxu](https://github.com/pacoxu)) +- Fixes add-on manager leader election to use leases instead of endpoints, similar to what kube-controller-manager does in 1.20 ([#98968](https://github.com/kubernetes/kubernetes/pull/98968), [@liggitt](https://github.com/liggitt)) +- Fixes connection errors when using `--volume-host-cidr-denylist` or `--volume-host-allow-local-loopback` ([#98436](https://github.com/kubernetes/kubernetes/pull/98436), [@liggitt](https://github.com/liggitt)) [SIG Network and Storage] +- Fixes problem where invalid selector on `PodDisruptionBudget` leads to a nil pointer dereference that causes the Controller manager to crash loop. ([#98750](https://github.com/kubernetes/kubernetes/pull/98750), [@mortent](https://github.com/mortent)) +- Fixes spurious errors about IPv6 in `kube-proxy` logs on nodes with IPv6 disabled. ([#99127](https://github.com/kubernetes/kubernetes/pull/99127), [@danwinship](https://github.com/danwinship)) +- Fixing a bug where a failed node may not have the NoExecute taint set correctly ([#96876](https://github.com/kubernetes/kubernetes/pull/96876), [@howieyuen](https://github.com/howieyuen)) [SIG Apps and Node] +- GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. ([#97740](https://github.com/kubernetes/kubernetes/pull/97740), [@prameshj](https://github.com/prameshj)) [SIG Cloud Provider and Network] +- Ignore update pod with no new images in alwaysPullImages admission controller ([#96668](https://github.com/kubernetes/kubernetes/pull/96668), [@pacoxu](https://github.com/pacoxu)) [SIG Apps, Auth and Node] +- Improve speed of vSphere PV provisioning and reduce number of API calls ([#100054](https://github.com/kubernetes/kubernetes/pull/100054), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] +- KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. ([#98158](https://github.com/kubernetes/kubernetes/pull/98158), [@dougsland](https://github.com/dougsland)) [SIG CLI] +- Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume ([#99446](https://github.com/kubernetes/kubernetes/pull/99446), [@pohly](https://github.com/pohly)) [SIG Apps, Node and Storage] +- Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS ([#97336](https://github.com/kubernetes/kubernetes/pull/97336), [@maaoBit](https://github.com/maaoBit)) [SIG Network] +- Kubeadm installs etcd v3.4.13 when creating cluster v1.19 ([#97244](https://github.com/kubernetes/kubernetes/pull/97244), [@pacoxu](https://github.com/pacoxu)) +- Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. ([#97016](https://github.com/kubernetes/kubernetes/pull/97016), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] +- Kubeadm: Some text in the `kubeadm upgrade plan` output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. ([#98728](https://github.com/kubernetes/kubernetes/pull/98728), [@stmcginnis](https://github.com/stmcginnis)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms ([#97403](https://github.com/kubernetes/kubernetes/pull/97403), [@abelbarrera15](https://github.com/abelbarrera15)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. ([#97372](https://github.com/kubernetes/kubernetes/pull/97372), [@ihgann](https://github.com/ihgann)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. ([#97919](https://github.com/kubernetes/kubernetes/pull/97919), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. ([#98882](https://github.com/kubernetes/kubernetes/pull/98882), [@kvaps](https://github.com/kvaps)) [SIG Cluster Lifecycle] +- Kubeadm: get k8s CI version markers from k8s infra bucket ([#98836](https://github.com/kubernetes/kubernetes/pull/98836), [@hasheddan](https://github.com/hasheddan)) [SIG Cluster Lifecycle and Release] +- Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false ([#98984](https://github.com/kubernetes/kubernetes/pull/98984), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] +- Kubectl logs: `--ignore-errors` is now honored by all containers, maintaining consistency with parallelConsumeRequest behavior. ([#97686](https://github.com/kubernetes/kubernetes/pull/97686), [@wzshiming](https://github.com/wzshiming)) +- Kubectl-convert: Fix `no kind "Ingress" is registered for version` error ([#97754](https://github.com/kubernetes/kubernetes/pull/97754), [@wzshiming](https://github.com/wzshiming)) +- Kubectl: Fixed panic when describing an ingress backend without an API Group ([#100505](https://github.com/kubernetes/kubernetes/pull/100505), [@lauchokyip](https://github.com/lauchokyip)) [SIG CLI] +- Kubelet now cleans up orphaned volume directories automatically ([#95301](https://github.com/kubernetes/kubernetes/pull/95301), [@lorenz](https://github.com/lorenz)) [SIG Node and Storage] +- Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. ([#96616](https://github.com/kubernetes/kubernetes/pull/96616), [@perithompson](https://github.com/perithompson)) [SIG Node and Windows] +- Kubelet: Fix kubelet from panic after getting the wrong signal ([#98200](https://github.com/kubernetes/kubernetes/pull/98200), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Kubelet: Fix repeatedly acquiring the inhibit lock ([#98088](https://github.com/kubernetes/kubernetes/pull/98088), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows ([#97378](https://github.com/kubernetes/kubernetes/pull/97378), [@hwdef](https://github.com/hwdef)) [SIG Node and Windows] +- Limits lease to have 1000 maximum attached objects. ([#98257](https://github.com/kubernetes/kubernetes/pull/98257), [@lingsamuel](https://github.com/lingsamuel)) +- Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. ([#97934](https://github.com/kubernetes/kubernetes/pull/97934), [@mattcary](https://github.com/mattcary)) [SIG Cloud Provider and Storage] +- On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report `ipFamilyPolicy RequireDualStack` and will have entries in `ipFamilies[]` for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. ([#99555](https://github.com/kubernetes/kubernetes/pull/99555), [@thockin](https://github.com/thockin)) [SIG Apps and Network] +- Performance regression #97685 has been fixed. ([#97860](https://github.com/kubernetes/kubernetes/pull/97860), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] +- Pod Log stats for windows now reports metrics ([#99221](https://github.com/kubernetes/kubernetes/pull/99221), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node, Storage, Testing and Windows] +- Pod status updates faster when reacting on probe results. The first readiness probe will be called faster when startup probes succeeded, which will make Pod status as ready faster. ([#98376](https://github.com/kubernetes/kubernetes/pull/98376), [@matthyx](https://github.com/matthyx)) +- Readjust `kubelet_containers_per_pod_count` buckets to only show metrics greater than 1. ([#98169](https://github.com/kubernetes/kubernetes/pull/98169), [@wawa0210](https://github.com/wawa0210)) +- Remove CSI topology from migrated in-tree gcepd volume. ([#97823](https://github.com/kubernetes/kubernetes/pull/97823), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider and Storage] +- Requests with invalid timeout parameters in the request URL now appear in the audit log correctly. ([#96901](https://github.com/kubernetes/kubernetes/pull/96901), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing] +- Resolve a "concurrent map read and map write" crashing error in the kubelet ([#95111](https://github.com/kubernetes/kubernetes/pull/95111), [@choury](https://github.com/choury)) [SIG Node] +- Resolves spurious `Failed to list *v1.Secret` or `Failed to list *v1.ConfigMap` messages in kubelet logs. ([#99538](https://github.com/kubernetes/kubernetes/pull/99538), [@liggitt](https://github.com/liggitt)) [SIG Auth and Node] +- ResourceQuota of an entity now inclusively calculate Pod overhead ([#99600](https://github.com/kubernetes/kubernetes/pull/99600), [@gjkim42](https://github.com/gjkim42)) +- Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using `dockershim` as a runtime. ([#99585](https://github.com/kubernetes/kubernetes/pull/99585), [@Iceber](https://github.com/Iceber)) +- Reverts breaking change to inline AzureFile volumes; referenced secrets are now searched for in the same namespace as the pod as in previous releases. ([#100563](https://github.com/kubernetes/kubernetes/pull/100563), [@msau42](https://github.com/msau42)) +- Scores from InterPodAffinity have stronger differentiation. ([#98096](https://github.com/kubernetes/kubernetes/pull/98096), [@leileiwan](https://github.com/leileiwan)) [SIG Scheduling] +- Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. ([#93510](https://github.com/kubernetes/kubernetes/pull/93510), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] +- Static pods will be deleted gracefully. ([#98103](https://github.com/kubernetes/kubernetes/pull/98103), [@gjkim42](https://github.com/gjkim42)) [SIG Node] +- Sync node status during kubelet node shutdown. + Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. ([#98005](https://github.com/kubernetes/kubernetes/pull/98005), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- The calculation of pod UIDs for static pods has changed to ensure each static pod gets a unique value - this will cause all static pod containers to be recreated/restarted if an in-place kubelet upgrade from 1.20 to 1.21 is performed. Note that draining pods before upgrading the kubelet across minor versions is the supported upgrade path. ([#87461](https://github.com/kubernetes/kubernetes/pull/87461), [@bboreham](https://github.com/bboreham)) [SIG Node] +- The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 ([#99795](https://github.com/kubernetes/kubernetes/pull/99795), [@robscott](https://github.com/robscott)) [SIG Network] +- Truncates a message if it hits the `NoteLengthLimit` when the scheduler records an event for the pod that indicates the pod has failed to schedule. ([#98715](https://github.com/kubernetes/kubernetes/pull/98715), [@carlory](https://github.com/carlory)) +- Updated k8s.gcr.io/ingress-gce-404-server-with-metrics-amd64 to a version that serves /metrics endpoint on a non-default port. ([#97621](https://github.com/kubernetes/kubernetes/pull/97621), [@vbannai](https://github.com/vbannai)) [SIG Cloud Provider] +- Updates the commands ` + - kubectl kustomize {arg} + - kubectl apply -k {arg} + `to use same code as kustomize CLI [v4.0.5](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.0.5) ([#98946](https://github.com/kubernetes/kubernetes/pull/98946), [@monopole](https://github.com/monopole)) +- Use force unmount for NFS volumes if regular mount fails after 1 minute timeout ([#96844](https://github.com/kubernetes/kubernetes/pull/96844), [@gnufied](https://github.com/gnufied)) [SIG Storage] +- Use network.Interface.VirtualMachine.ID to get the binded VM + Skip standalone VM when reconciling LoadBalancer ([#97635](https://github.com/kubernetes/kubernetes/pull/97635), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. ([#97857](https://github.com/kubernetes/kubernetes/pull/97857), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Auth] +- When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the + "primary IP family" of the cluster by picking a primary pod IP of the same family + as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs + in the order the administrator wanted (since some CNI plugins don't allow + configuring this). ([#97979](https://github.com/kubernetes/kubernetes/pull/97979), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] +- When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. ([#99122](https://github.com/kubernetes/kubernetes/pull/99122), [@huffmanca](https://github.com/huffmanca)) [SIG Cloud Provider and Storage] +- When using `Containerd` on Windows, the `C:\Windows\System32\drivers\etc\hosts` file will now be managed by kubelet. ([#83730](https://github.com/kubernetes/kubernetes/pull/83730), [@claudiubelu](https://github.com/claudiubelu)) +- `VolumeBindingArgs` now allow `BindTimeoutSeconds` to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. ([#99835](https://github.com/kubernetes/kubernetes/pull/99835), [@chendave](https://github.com/chendave)) [SIG Scheduling and Storage] +- `kubectl exec` and `kubectl attach` now honor the `--quiet` flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. ([#99004](https://github.com/kubernetes/kubernetes/pull/99004), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] ### Other (Cleanup or Flake) -- Handle slow cronjob lister in cronjob controller v2 and improve memory footprint. ([#96443](https://github.com/kubernetes/kubernetes/pull/96443), [@alaypatel07](https://github.com/alaypatel07)) [SIG Apps] -- --redirect-container-streaming is no longer functional. The flag will be removed in v1.22 ([#95935](https://github.com/kubernetes/kubernetes/pull/95935), [@tallclair](https://github.com/tallclair)) [SIG Node] -- A new metric `requestAbortsTotal` has been introduced that counts aborted requests for each `group`, `version`, `verb`, `resource`, `subresource` and `scope`. ([#95002](https://github.com/kubernetes/kubernetes/pull/95002), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery, Cloud Provider, Instrumentation and Scheduling] -- API priority and fairness metrics use snake_case in label names ([#96236](https://github.com/kubernetes/kubernetes/pull/96236), [@adtac](https://github.com/adtac)) [SIG API Machinery, Cluster Lifecycle, Instrumentation and Testing] -- Applies translations on all command descriptions ([#95439](https://github.com/kubernetes/kubernetes/pull/95439), [@HerrNaN](https://github.com/HerrNaN)) [SIG CLI] -- Changed: default "Accept-Encoding" header removed from HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes ([#96127](https://github.com/kubernetes/kubernetes/pull/96127), [@fonsecas72](https://github.com/fonsecas72)) [SIG Network and Node] -- Generators for services are removed from kubectl ([#95256](https://github.com/kubernetes/kubernetes/pull/95256), [@Git-Jiro](https://github.com/Git-Jiro)) [SIG CLI] -- Introduce kubectl-convert plugin. ([#96190](https://github.com/kubernetes/kubernetes/pull/96190), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing] -- Kube-scheduler now logs processed component config at startup ([#96426](https://github.com/kubernetes/kubernetes/pull/96426), [@damemi](https://github.com/damemi)) [SIG Scheduling] -- NONE ([#96179](https://github.com/kubernetes/kubernetes/pull/96179), [@bbyrne5](https://github.com/bbyrne5)) [SIG Network] -- Users will now be able to configure all supported values for AWS NLB health check interval and thresholds for new resources. ([#96312](https://github.com/kubernetes/kubernetes/pull/96312), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] +- APIs for kubelet annotations and labels from `k8s.io/kubernetes/pkg/kubelet/apis` are now moved under `k8s.io/kubelet/pkg/apis/` ([#98931](https://github.com/kubernetes/kubernetes/pull/98931), [@michaelbeaumont](https://github.com/michaelbeaumont)) +- Apiserver_request_duration_seconds is promoted to stable status. ([#99925](https://github.com/kubernetes/kubernetes/pull/99925), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] +- Bump github.com/Azure/go-autorest/autorest to v0.11.12 ([#97033](https://github.com/kubernetes/kubernetes/pull/97033), [@patrickshan](https://github.com/patrickshan)) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle] +- Clients required to use go1.15.8+ or go1.16+ if kube-apiserver has the goaway feature enabled to avoid unexpected data race condition. ([#98809](https://github.com/kubernetes/kubernetes/pull/98809), [@answer1991](https://github.com/answer1991)) +- Delete deprecated `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols` mixed procotol annotation in favor of the MixedProtocolLBService feature ([#97096](https://github.com/kubernetes/kubernetes/pull/97096), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- EndpointSlice generation is now incremented when labels change. ([#99750](https://github.com/kubernetes/kubernetes/pull/99750), [@robscott](https://github.com/robscott)) [SIG Network] +- Featuregate AllowInsecureBackendProxy graduates to GA and unconditionally enabled. ([#99658](https://github.com/kubernetes/kubernetes/pull/99658), [@deads2k](https://github.com/deads2k)) +- Increase timeout for pod lifecycle test to reach pod status=ready ([#96691](https://github.com/kubernetes/kubernetes/pull/96691), [@hh](https://github.com/hh)) +- Increased `CSINodeIDMaxLength` from 128 bytes to 192 bytes. ([#98753](https://github.com/kubernetes/kubernetes/pull/98753), [@Jiawei0227](https://github.com/Jiawei0227)) +- Kube-apiserver: The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. ([#97693](https://github.com/kubernetes/kubernetes/pull/97693), [@enj](https://github.com/enj)) [SIG API Machinery and Auth] +- Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. ([#96296](https://github.com/kubernetes/kubernetes/pull/96296), [@aojea](https://github.com/aojea)) [SIG Network and Testing] +- Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' ([#97087](https://github.com/kubernetes/kubernetes/pull/97087), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] +- Kubectl: The deprecated `kubectl alpha debug` command is removed. Use `kubectl debug` instead. ([#98111](https://github.com/kubernetes/kubernetes/pull/98111), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] +- Kubelet command line flags related to dockershim are now showing deprecation message as they will be removed along with dockershim in future release. ([#98730](https://github.com/kubernetes/kubernetes/pull/98730), [@dims](https://github.com/dims)) +- Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97618](https://github.com/kubernetes/kubernetes/pull/97618), [@jherrera123](https://github.com/jherrera123)) [SIG Release and Testing] +- Process start time on Windows now uses current process information ([#97491](https://github.com/kubernetes/kubernetes/pull/97491), [@jsturtevant](https://github.com/jsturtevant)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows] +- Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object ([#98430](https://github.com/kubernetes/kubernetes/pull/98430), [@liggitt](https://github.com/liggitt)) [SIG Network and Testing] +- The `AttachVolumeLimit` feature gate (GA since v1.17) has been removed and now unconditionally enabled. ([#96539](https://github.com/kubernetes/kubernetes/pull/96539), [@ialidzhikov](https://github.com/ialidzhikov)) +- The `CSINodeInfo` feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the `--feature-gates` argument. ([#96561](https://github.com/kubernetes/kubernetes/pull/96561), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Apps, Auth, Scheduling, Storage and Testing] +- The `apiserver_request_total` metric is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. ([#99788](https://github.com/kubernetes/kubernetes/pull/99788), [@logicalhan](https://github.com/logicalhan)) +- The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. ([#98325](https://github.com/kubernetes/kubernetes/pull/98325), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth, Cloud Provider and Scheduling] +- The deprecated feature gates `CSIDriverRegistry`, `BlockVolume` and `CSIBlockVolume` are now unconditionally enabled and can no longer be specified in component invocations. ([#98021](https://github.com/kubernetes/kubernetes/pull/98021), [@gavinfish](https://github.com/gavinfish)) [SIG Storage] +- The deprecated feature gates `RotateKubeletClientCertificate`, `AttachVolumeLimit`, `VolumePVCDataSource` and `EvenPodsSpread` are now unconditionally enabled and can no longer be specified in component invocations. ([#97306](https://github.com/kubernetes/kubernetes/pull/97306), [@gavinfish](https://github.com/gavinfish)) [SIG Node, Scheduling and Storage] +- The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing `--allowed-not-ready-nodes=-1` when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. ([#98781](https://github.com/kubernetes/kubernetes/pull/98781), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] +- The feature gates `WindowsGMSA` and `WindowsRunAsUserName` that are GA since v1.18 are now removed. ([#96531](https://github.com/kubernetes/kubernetes/pull/96531), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Node and Windows] +- The new `-gce-zones` flag on the `e2e.test` binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. ([#98787](https://github.com/kubernetes/kubernetes/pull/98787), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Cluster Lifecycle and Testing] +- Update cri-tools to [v1.20.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.20.0) ([#97967](https://github.com/kubernetes/kubernetes/pull/97967), [@rajibmitra](https://github.com/rajibmitra)) [SIG Cloud Provider] +- Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. ([#98284](https://github.com/kubernetes/kubernetes/pull/98284), [@pjh](https://github.com/pjh)) [SIG Cloud Provider] +- `apiserver_storage_objects` (a newer version of `etcd_object_counts`) is promoted and marked as stable. ([#100082](https://github.com/kubernetes/kubernetes/pull/100082), [@logicalhan](https://github.com/logicalhan)) + +### Uncategorized + +- GCE L4 Loadbalancers now handle > 5 ports in service spec correctly. ([#99595](https://github.com/kubernetes/kubernetes/pull/99595), [@prameshj](https://github.com/prameshj)) [SIG Cloud Provider] +- The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. ([#99610](https://github.com/kubernetes/kubernetes/pull/99610), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] ## Dependencies ### Added -- cloud.google.com/go/firestore: v1.1.0 -- github.com/armon/go-metrics: [f0300d1](https://github.com/armon/go-metrics/tree/f0300d1) -- github.com/armon/go-radix: [7fddfc3](https://github.com/armon/go-radix/tree/7fddfc3) -- github.com/bketelsen/crypt: [5cbc8cc](https://github.com/bketelsen/crypt/tree/5cbc8cc) -- github.com/hashicorp/consul/api: [v1.1.0](https://github.com/hashicorp/consul/api/tree/v1.1.0) -- github.com/hashicorp/consul/sdk: [v0.1.1](https://github.com/hashicorp/consul/sdk/tree/v0.1.1) -- github.com/hashicorp/errwrap: [v1.0.0](https://github.com/hashicorp/errwrap/tree/v1.0.0) -- github.com/hashicorp/go-cleanhttp: [v0.5.1](https://github.com/hashicorp/go-cleanhttp/tree/v0.5.1) -- github.com/hashicorp/go-immutable-radix: [v1.0.0](https://github.com/hashicorp/go-immutable-radix/tree/v1.0.0) -- github.com/hashicorp/go-msgpack: [v0.5.3](https://github.com/hashicorp/go-msgpack/tree/v0.5.3) -- github.com/hashicorp/go-multierror: [v1.0.0](https://github.com/hashicorp/go-multierror/tree/v1.0.0) -- github.com/hashicorp/go-rootcerts: [v1.0.0](https://github.com/hashicorp/go-rootcerts/tree/v1.0.0) -- github.com/hashicorp/go-sockaddr: [v1.0.0](https://github.com/hashicorp/go-sockaddr/tree/v1.0.0) -- github.com/hashicorp/go-uuid: [v1.0.1](https://github.com/hashicorp/go-uuid/tree/v1.0.1) -- github.com/hashicorp/go.net: [v0.0.1](https://github.com/hashicorp/go.net/tree/v0.0.1) -- github.com/hashicorp/logutils: [v1.0.0](https://github.com/hashicorp/logutils/tree/v1.0.0) -- github.com/hashicorp/mdns: [v1.0.0](https://github.com/hashicorp/mdns/tree/v1.0.0) -- github.com/hashicorp/memberlist: [v0.1.3](https://github.com/hashicorp/memberlist/tree/v0.1.3) -- github.com/hashicorp/serf: [v0.8.2](https://github.com/hashicorp/serf/tree/v0.8.2) -- github.com/mitchellh/cli: [v1.0.0](https://github.com/mitchellh/cli/tree/v1.0.0) -- github.com/mitchellh/go-testing-interface: [v1.0.0](https://github.com/mitchellh/go-testing-interface/tree/v1.0.0) -- github.com/mitchellh/gox: [v0.4.0](https://github.com/mitchellh/gox/tree/v0.4.0) -- github.com/mitchellh/iochan: [v1.0.0](https://github.com/mitchellh/iochan/tree/v1.0.0) -- github.com/pascaldekloe/goe: [57f6aae](https://github.com/pascaldekloe/goe/tree/57f6aae) -- github.com/posener/complete: [v1.1.1](https://github.com/posener/complete/tree/v1.1.1) -- github.com/ryanuber/columnize: [9b3edd6](https://github.com/ryanuber/columnize/tree/9b3edd6) -- github.com/sean-/seed: [e2103e2](https://github.com/sean-/seed/tree/e2103e2) -- github.com/subosito/gotenv: [v1.2.0](https://github.com/subosito/gotenv/tree/v1.2.0) -- github.com/willf/bitset: [d5bec33](https://github.com/willf/bitset/tree/d5bec33) -- gopkg.in/ini.v1: v1.51.0 -- gopkg.in/yaml.v3: 9f266ea +- github.com/go-errors/errors: [v1.0.1](https://github.com/go-errors/errors/tree/v1.0.1) +- github.com/gobuffalo/here: [v0.6.0](https://github.com/gobuffalo/here/tree/v0.6.0) +- github.com/google/shlex: [e7afc7f](https://github.com/google/shlex/tree/e7afc7f) +- github.com/markbates/pkger: [v0.17.1](https://github.com/markbates/pkger/tree/v0.17.1) +- github.com/moby/spdystream: [v0.2.0](https://github.com/moby/spdystream/tree/v0.2.0) +- github.com/monochromegane/go-gitignore: [205db1a](https://github.com/monochromegane/go-gitignore/tree/205db1a) +- github.com/niemeyer/pretty: [a10e7ca](https://github.com/niemeyer/pretty/tree/a10e7ca) +- github.com/xlab/treeprint: [a009c39](https://github.com/xlab/treeprint/tree/a009c39) +- go.starlark.net: 8dd3e2e +- golang.org/x/term: 6a3ed07 +- sigs.k8s.io/kustomize/api: v0.8.5 +- sigs.k8s.io/kustomize/cmd/config: v0.9.7 +- sigs.k8s.io/kustomize/kustomize/v4: v4.0.5 +- sigs.k8s.io/kustomize/kyaml: v0.10.15 + +### Changed +- dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891 +- github.com/Azure/go-autorest/autorest: [v0.11.1 → v0.11.12](https://github.com/Azure/go-autorest/autorest/compare/v0.11.1...v0.11.12) +- github.com/NYTimes/gziphandler: [56545f4 → v1.1.1](https://github.com/NYTimes/gziphandler/compare/56545f4...v1.1.1) +- github.com/cilium/ebpf: [1c8d4c9 → v0.2.0](https://github.com/cilium/ebpf/compare/1c8d4c9...v0.2.0) +- github.com/container-storage-interface/spec: [v1.2.0 → v1.3.0](https://github.com/container-storage-interface/spec/compare/v1.2.0...v1.3.0) +- github.com/containerd/console: [v1.0.0 → v1.0.1](https://github.com/containerd/console/compare/v1.0.0...v1.0.1) +- github.com/containerd/containerd: [v1.4.1 → v1.4.4](https://github.com/containerd/containerd/compare/v1.4.1...v1.4.4) +- github.com/coredns/corefile-migration: [v1.0.10 → v1.0.11](https://github.com/coredns/corefile-migration/compare/v1.0.10...v1.0.11) +- github.com/creack/pty: [v1.1.7 → v1.1.11](https://github.com/creack/pty/compare/v1.1.7...v1.1.11) +- github.com/docker/docker: [bd33bbf → v20.10.2+incompatible](https://github.com/docker/docker/compare/bd33bbf...v20.10.2) +- github.com/go-logr/logr: [v0.2.0 → v0.4.0](https://github.com/go-logr/logr/compare/v0.2.0...v0.4.0) +- github.com/go-openapi/spec: [v0.19.3 → v0.19.5](https://github.com/go-openapi/spec/compare/v0.19.3...v0.19.5) +- github.com/go-openapi/strfmt: [v0.19.3 → v0.19.5](https://github.com/go-openapi/strfmt/compare/v0.19.3...v0.19.5) +- github.com/go-openapi/validate: [v0.19.5 → v0.19.8](https://github.com/go-openapi/validate/compare/v0.19.5...v0.19.8) +- github.com/gogo/protobuf: [v1.3.1 → v1.3.2](https://github.com/gogo/protobuf/compare/v1.3.1...v1.3.2) +- github.com/golang/mock: [v1.4.1 → v1.4.4](https://github.com/golang/mock/compare/v1.4.1...v1.4.4) +- github.com/google/cadvisor: [v0.38.5 → v0.39.0](https://github.com/google/cadvisor/compare/v0.38.5...v0.39.0) +- github.com/heketi/heketi: [c2e2a4a → v10.2.0+incompatible](https://github.com/heketi/heketi/compare/c2e2a4a...v10.2.0) +- github.com/kisielk/errcheck: [v1.2.0 → v1.5.0](https://github.com/kisielk/errcheck/compare/v1.2.0...v1.5.0) +- github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.2](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.2) +- github.com/kr/text: [v0.1.0 → v0.2.0](https://github.com/kr/text/compare/v0.1.0...v0.2.0) +- github.com/mattn/go-runewidth: [v0.0.2 → v0.0.7](https://github.com/mattn/go-runewidth/compare/v0.0.2...v0.0.7) +- github.com/miekg/dns: [v1.1.4 → v1.1.35](https://github.com/miekg/dns/compare/v1.1.4...v1.1.35) +- github.com/moby/sys/mountinfo: [v0.1.3 → v0.4.0](https://github.com/moby/sys/mountinfo/compare/v0.1.3...v0.4.0) +- github.com/moby/term: [672ec06 → df9cb8a](https://github.com/moby/term/compare/672ec06...df9cb8a) +- github.com/mrunalp/fileutils: [abd8a0e → v0.5.0](https://github.com/mrunalp/fileutils/compare/abd8a0e...v0.5.0) +- github.com/olekukonko/tablewriter: [a0225b3 → v0.0.4](https://github.com/olekukonko/tablewriter/compare/a0225b3...v0.0.4) +- github.com/opencontainers/runc: [v1.0.0-rc92 → v1.0.0-rc93](https://github.com/opencontainers/runc/compare/v1.0.0-rc92...v1.0.0-rc93) +- github.com/opencontainers/runtime-spec: [4d89ac9 → e6143ca](https://github.com/opencontainers/runtime-spec/compare/4d89ac9...e6143ca) +- github.com/opencontainers/selinux: [v1.6.0 → v1.8.0](https://github.com/opencontainers/selinux/compare/v1.6.0...v1.8.0) +- github.com/sergi/go-diff: [v1.0.0 → v1.1.0](https://github.com/sergi/go-diff/compare/v1.0.0...v1.1.0) +- github.com/sirupsen/logrus: [v1.6.0 → v1.7.0](https://github.com/sirupsen/logrus/compare/v1.6.0...v1.7.0) +- github.com/syndtr/gocapability: [d983527 → 42c35b4](https://github.com/syndtr/gocapability/compare/d983527...42c35b4) +- github.com/willf/bitset: [d5bec33 → v1.1.11](https://github.com/willf/bitset/compare/d5bec33...v1.1.11) +- github.com/yuin/goldmark: [v1.1.27 → v1.2.1](https://github.com/yuin/goldmark/compare/v1.1.27...v1.2.1) +- golang.org/x/crypto: 7f63de1 → 5ea612d +- golang.org/x/exp: 6cc2880 → 85be41e +- golang.org/x/mobile: d2bd2a2 → e6ae53a +- golang.org/x/mod: v0.3.0 → ce943fd +- golang.org/x/net: 69a7880 → 3d97a24 +- golang.org/x/sync: cd5d95a → 67f06af +- golang.org/x/sys: 5cba982 → a50acf3 +- golang.org/x/time: 3af7569 → f8bda1e +- golang.org/x/tools: c1934b7 → v0.1.0 +- gopkg.in/check.v1: 41f04d3 → 8fa4692 +- gopkg.in/yaml.v2: v2.2.8 → v2.4.0 +- gotest.tools/v3: v3.0.2 → v3.0.3 +- k8s.io/gengo: 83324d8 → b6c5ce2 +- k8s.io/klog/v2: v2.4.0 → v2.8.0 +- k8s.io/kube-openapi: d219536 → 591a79e +- k8s.io/system-validators: v1.2.0 → v1.4.0 +- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15 +- sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.1.0 + +### Removed +- github.com/codegangsta/negroni: [v1.0.0](https://github.com/codegangsta/negroni/tree/v1.0.0) +- github.com/docker/spdystream: [449fdfc](https://github.com/docker/spdystream/tree/449fdfc) +- github.com/golangplus/bytes: [45c989f](https://github.com/golangplus/bytes/tree/45c989f) +- github.com/golangplus/fmt: [2a5d6d7](https://github.com/golangplus/fmt/tree/2a5d6d7) +- github.com/gorilla/context: [v1.1.1](https://github.com/gorilla/context/tree/v1.1.1) +- github.com/kr/pty: [v1.1.5](https://github.com/kr/pty/tree/v1.1.5) - rsc.io/quote/v3: v3.1.0 - rsc.io/sampler: v1.3.0 +- sigs.k8s.io/kustomize: v2.0.3+incompatible -### Changed -- cloud.google.com/go/bigquery: v1.0.1 → v1.4.0 -- cloud.google.com/go/datastore: v1.0.0 → v1.1.0 -- cloud.google.com/go/pubsub: v1.0.1 → v1.2.0 -- cloud.google.com/go/storage: v1.0.0 → v1.6.0 -- cloud.google.com/go: v0.51.0 → v0.54.0 -- github.com/Microsoft/go-winio: [fc70bd9 → v0.4.15](https://github.com/Microsoft/go-winio/compare/fc70bd9...v0.4.15) -- github.com/aws/aws-sdk-go: [v1.35.5 → v1.35.24](https://github.com/aws/aws-sdk-go/compare/v1.35.5...v1.35.24) -- github.com/blang/semver: [v3.5.0+incompatible → v3.5.1+incompatible](https://github.com/blang/semver/compare/v3.5.0...v3.5.1) -- github.com/checkpoint-restore/go-criu/v4: [v4.0.2 → v4.1.0](https://github.com/checkpoint-restore/go-criu/v4/compare/v4.0.2...v4.1.0) -- github.com/containerd/containerd: [v1.3.3 → v1.4.1](https://github.com/containerd/containerd/compare/v1.3.3...v1.4.1) -- github.com/containerd/ttrpc: [v1.0.0 → v1.0.2](https://github.com/containerd/ttrpc/compare/v1.0.0...v1.0.2) -- github.com/containerd/typeurl: [v1.0.0 → v1.0.1](https://github.com/containerd/typeurl/compare/v1.0.0...v1.0.1) -- github.com/coreos/etcd: [v3.3.10+incompatible → v3.3.13+incompatible](https://github.com/coreos/etcd/compare/v3.3.10...v3.3.13) -- github.com/docker/docker: [aa6a989 → bd33bbf](https://github.com/docker/docker/compare/aa6a989...bd33bbf) -- github.com/go-gl/glfw/v3.3/glfw: [12ad95a → 6f7a984](https://github.com/go-gl/glfw/v3.3/glfw/compare/12ad95a...6f7a984) -- github.com/golang/groupcache: [215e871 → 8c9f03a](https://github.com/golang/groupcache/compare/215e871...8c9f03a) -- github.com/golang/mock: [v1.3.1 → v1.4.1](https://github.com/golang/mock/compare/v1.3.1...v1.4.1) -- github.com/golang/protobuf: [v1.4.2 → v1.4.3](https://github.com/golang/protobuf/compare/v1.4.2...v1.4.3) -- github.com/google/cadvisor: [v0.37.0 → v0.38.4](https://github.com/google/cadvisor/compare/v0.37.0...v0.38.4) -- github.com/google/go-cmp: [v0.4.0 → v0.5.2](https://github.com/google/go-cmp/compare/v0.4.0...v0.5.2) -- github.com/google/pprof: [d4f498a → 1ebb73c](https://github.com/google/pprof/compare/d4f498a...1ebb73c) -- github.com/google/uuid: [v1.1.1 → v1.1.2](https://github.com/google/uuid/compare/v1.1.1...v1.1.2) -- github.com/gorilla/mux: [v1.7.3 → v1.8.0](https://github.com/gorilla/mux/compare/v1.7.3...v1.8.0) -- github.com/gorilla/websocket: [v1.4.0 → v1.4.2](https://github.com/gorilla/websocket/compare/v1.4.0...v1.4.2) -- github.com/karrick/godirwalk: [v1.7.5 → v1.16.1](https://github.com/karrick/godirwalk/compare/v1.7.5...v1.16.1) -- github.com/opencontainers/runc: [819fcc6 → v1.0.0-rc92](https://github.com/opencontainers/runc/compare/819fcc6...v1.0.0-rc92) -- github.com/opencontainers/runtime-spec: [237cc4f → 4d89ac9](https://github.com/opencontainers/runtime-spec/compare/237cc4f...4d89ac9) -- github.com/opencontainers/selinux: [v1.5.2 → v1.6.0](https://github.com/opencontainers/selinux/compare/v1.5.2...v1.6.0) -- github.com/prometheus/procfs: [v0.1.3 → v0.2.0](https://github.com/prometheus/procfs/compare/v0.1.3...v0.2.0) -- github.com/quobyte/api: [v0.1.2 → v0.1.8](https://github.com/quobyte/api/compare/v0.1.2...v0.1.8) -- github.com/spf13/cobra: [v1.0.0 → v1.1.1](https://github.com/spf13/cobra/compare/v1.0.0...v1.1.1) -- github.com/spf13/viper: [v1.4.0 → v1.7.0](https://github.com/spf13/viper/compare/v1.4.0...v1.7.0) -- github.com/stretchr/testify: [v1.4.0 → v1.6.1](https://github.com/stretchr/testify/compare/v1.4.0...v1.6.1) -- github.com/vishvananda/netns: [52d707b → db3c7e5](https://github.com/vishvananda/netns/compare/52d707b...db3c7e5) -- go.opencensus.io: v0.22.2 → v0.22.3 -- golang.org/x/exp: da58074 → 6cc2880 -- golang.org/x/lint: fdd1cda → 738671d -- golang.org/x/net: ab34263 → 69a7880 -- golang.org/x/oauth2: 858c2ad → bf48bf1 -- golang.org/x/sys: ed371f2 → 5cba982 -- golang.org/x/text: v0.3.3 → v0.3.4 -- golang.org/x/time: 555d28b → 3af7569 -- golang.org/x/xerrors: 9bdfabe → 5ec99f8 -- google.golang.org/api: v0.15.1 → v0.20.0 -- google.golang.org/genproto: cb27e3a → 8816d57 -- google.golang.org/grpc: v1.27.0 → v1.27.1 -- google.golang.org/protobuf: v1.24.0 → v1.25.0 -- honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3 -- k8s.io/gengo: 8167cfd → 83324d8 -- k8s.io/klog/v2: v2.2.0 → v2.4.0 -- k8s.io/kube-openapi: 8b50664 → d219536 -- k8s.io/utils: d5654de → 67b214c -- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.12 → v0.0.14 -- sigs.k8s.io/structured-merge-diff/v4: b3cf1e8 → v4.0.2 - -### Removed -- github.com/armon/consul-api: [eb2c6b5](https://github.com/armon/consul-api/tree/eb2c6b5) -- github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) -- github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) -- github.com/xordataexchange/crypt: [b2862e3](https://github.com/xordataexchange/crypt/tree/b2862e3) - - - -# v1.20.0-beta.1 - - -## Downloads for v1.20.0-beta.1 - -### Source Code - -filename | sha512 hash --------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes.tar.gz) | 4eddf4850c2d57751696f352d0667309339090aeb30ff93e8db8a22c6cdebf74cb2d5dc78d4ae384c4e25491efc39413e2e420a804b76b421a9ad934e56b0667 -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-src.tar.gz) | 59de5221162e9b6d88f5abbdb99765cb2b2e501498ea853fb65f2abe390211e28d9f21e0d87be3ade550a5ea6395d04552cf093d2ce2f99fd45ad46545dd13cb - -### Client binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-darwin-amd64.tar.gz) | d69ffed19b034a4221fc084e43ac293cf392e98febf5bf580f8d92307a8421d8b3aab18f9ca70608937e836b42c7a34e829f88eba6e040218a4486986e2fca21 -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-linux-386.tar.gz) | 1b542e165860c4adcd4550adc19b86c3db8cd75d2a1b8db17becc752da78b730ee48f1b0aaf8068d7bfbb1d8e023741ec293543bc3dd0f4037172a6917db8169 -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-linux-amd64.tar.gz) | 90ad52785eecb43a6f9035b92b6ba39fc84e67f8bc91cf098e70f8cfdd405c4b9d5c02dccb21022f21bb5b6ce92fdef304def1da0a7255c308e2c5fb3a9cdaab -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-linux-arm.tar.gz) | d0cb3322b056e1821679afa70728ffc0d3375e8f3326dabbe8185be2e60f665ab8985b13a1a432e10281b84a929e0f036960253ac0dd6e0b44677d539e98e61b -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-linux-arm64.tar.gz) | 3aecc8197e0aa368408624add28a2dd5e73f0d8a48e5e33c19edf91d5323071d16a27353a6f3e22df4f66ed7bfbae8e56e0a9050f7bbdf927ce6aeb29bba6374 -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-linux-ppc64le.tar.gz) | 6ff145058f62d478b98f1e418e272555bfb5c7861834fbbf10a8fb334cc7ff09b32f2666a54b230932ba71d2fc7d3b1c1f5e99e6fe6d6ec83926a9b931cd2474 -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-linux-s390x.tar.gz) | ff7b8bb894076e05a3524f6327a4a6353b990466f3292e84c92826cb64b5c82b3855f48b8e297ccadc8bcc15552bc056419ff6ff8725fc4e640828af9cc1331b -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-windows-386.tar.gz) | 6c6dcac9c725605763a130b5a975f2b560aa976a5c809d4e0887900701b707baccb9ca1aebc10a03cfa7338a6f42922bbf838ccf6800fc2a3e231686a72568b6 -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-client-windows-amd64.tar.gz) | d12e3a29c960f0ddd1b9aabf5426ac1259863ac6c8f2be1736ebeb57ddca6b1c747ee2c363be19e059e38cf71488c5ea3509ad4d0e67fd5087282a5ad0ae9a48 - -### Server binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-server-linux-amd64.tar.gz) | 904e8c049179e071c6caa65f525f465260bb4d4318a6dd9cc05be2172f39f7cfc69d1672736e01d926045764fe8872e806444e3af77ffef823ede769537b7d20 -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-server-linux-arm.tar.gz) | 5934959374868aed8d4294de84411972660bca7b2e952201a9403f37e40c60a5c53eaea8001344d0bf4a00c8cd27de6324d88161388de27f263a5761357cb82b -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-server-linux-arm64.tar.gz) | 4c884585970f80dc5462d9a734d7d5be9558b36c6e326a8a3139423efbd7284fa9f53fb077983647e17e19f03f5cb9bf26201450c78daecf10afa5a1ab5f9efc -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-server-linux-ppc64le.tar.gz) | 235b78b08440350dcb9f13b63f7722bd090c672d8e724ca5d409256e5a5d4f46d431652a1aa908c3affc5b1e162318471de443d38b93286113e79e7f90501a9b -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-server-linux-s390x.tar.gz) | 220fc9351702b3ecdcf79089892ceb26753a8a1deaf46922ffb3d3b62b999c93fef89440e779ca6043372b963081891b3a966d1a5df0cf261bdd44395fd28dce - -### Node binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-node-linux-amd64.tar.gz) | fe59d3a1f21c47bab126f689687657f77fbcb46a2caeef48eecd073b2b22879f997a466911b5c5c829e9cf27e68a36ecdf18686d42714839d4b97d6c7281578d -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-node-linux-arm.tar.gz) | 93e545aa963cfd11e0b2c6d47669b5ef70c5a86ef80c3353c1a074396bff1e8e7371dda25c39d78c7a9e761f2607b8b5ab843fa0c10b8ff9663098fae8d25725 -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-node-linux-arm64.tar.gz) | 5e0f177f9bec406a668d4b37e69b191208551fdf289c82b5ec898959da4f8a00a2b0695cbf1d2de5acb809321c6e5604f5483d33556543d92b96dcf80e814dd3 -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-node-linux-ppc64le.tar.gz) | 574412059e4d257eb904cd4892a075b6a2cde27adfa4976ee64c46d6768facece338475f1b652ad94c8df7cfcbb70ebdf0113be109c7099ab76ffdb6f023eefd -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-node-linux-s390x.tar.gz) | b1ffaa6d7f77d89885c642663cb14a86f3e2ec2afd223e3bb2000962758cf0f15320969ffc4be93b5826ff22d54fdbae0dbea09f9d8228eda6da50b6fdc88758 -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.1/kubernetes-node-windows-amd64.tar.gz) | 388983765213cf3bdc1f8b27103ed79e39028767e5f1571e35ed1f91ed100e49f3027f7b7ff19b53fab7fbb6d723c0439f21fc6ed62be64532c25f5bfa7ee265 - -## Changelog since v1.20.0-beta.0 - -## Changes by Kind - -### Deprecation - -- ACTION REQUIRED: The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags `--address` and `--insecure-bind-address` have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags `--port` and `--insecure-port` may only be set to 0 and will be removed in v1.24. ([#95856](https://github.com/kubernetes/kubernetes/pull/95856), [@knight42](https://github.com/knight42)) [SIG API Machinery, Node and Testing] - -### API Change - -- + `TokenRequest` and `TokenRequestProjection` features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The `TokenRequest` and `TokenRequestProjection` feature gates will be removed in v1.21. - + kubeadm's kube-apiserver Pod manifest now includes the following flags by default "--service-account-key-file", "--service-account-signing-key-file", "--service-account-issuer". ([#93258](https://github.com/kubernetes/kubernetes/pull/93258), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing] -- Certain fields on Service objects will be automatically cleared when changing the service's `type` to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. ([#95196](https://github.com/kubernetes/kubernetes/pull/95196), [@thockin](https://github.com/thockin)) [SIG API Machinery, Apps, Network and Testing] -- Services will now have a `clusterIPs` field to go with `clusterIP`. `clusterIPs[0]` is a synonym for `clusterIP` and will be syncronized on create and update operations. ([#95894](https://github.com/kubernetes/kubernetes/pull/95894), [@thockin](https://github.com/thockin)) [SIG Network] - -### Feature - -- A new metric `apiserver_request_filter_duration_seconds` has been introduced that - measures request filter latency in seconds. ([#95207](https://github.com/kubernetes/kubernetes/pull/95207), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Instrumentation] -- Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node there by disrupting kubelet process. ([#96051](https://github.com/kubernetes/kubernetes/pull/96051), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Node and Windows] -- Changed: default "Accept: */*" header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) ([#95641](https://github.com/kubernetes/kubernetes/pull/95641), [@fonsecas72](https://github.com/fonsecas72)) [SIG Network and Node] -- Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. ([#95489](https://github.com/kubernetes/kubernetes/pull/95489), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery and Auth] -- Kube-apiserver: added support for compressing rotated audit log files with `--audit-log-compress` ([#94066](https://github.com/kubernetes/kubernetes/pull/94066), [@lojies](https://github.com/lojies)) [SIG API Machinery and Auth] - -### Documentation - -- Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList ([#95117](https://github.com/kubernetes/kubernetes/pull/95117), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] - -### Bug or Regression - -- Added support to kube-proxy for externalTrafficPolicy=Local setting via Direct Server Return (DSR) load balancers on Windows. ([#93166](https://github.com/kubernetes/kubernetes/pull/93166), [@elweb9858](https://github.com/elweb9858)) [SIG Network] -- Disable watchcache for events ([#96052](https://github.com/kubernetes/kubernetes/pull/96052), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] -- Disabled `LocalStorageCapacityIsolation` feature gate is honored during scheduling. ([#96092](https://github.com/kubernetes/kubernetes/pull/96092), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] -- Fix bug in JSON path parser where an error occurs when a range is empty ([#95933](https://github.com/kubernetes/kubernetes/pull/95933), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] -- Fix k8s.io/apimachinery/pkg/api/meta.SetStatusCondition to update ObservedGeneration ([#95961](https://github.com/kubernetes/kubernetes/pull/95961), [@KnicKnic](https://github.com/KnicKnic)) [SIG API Machinery] -- Fixed a regression which prevented pods with `docker/default` seccomp annotations from being created in 1.19 if a PodSecurityPolicy was in place which did not allow `runtime/default` seccomp profiles. ([#95985](https://github.com/kubernetes/kubernetes/pull/95985), [@saschagrunert](https://github.com/saschagrunert)) [SIG Auth] -- Kubectl: print error if users place flags before plugin name ([#92343](https://github.com/kubernetes/kubernetes/pull/92343), [@knight42](https://github.com/knight42)) [SIG CLI] -- When creating a PVC with the volume.beta.kubernetes.io/storage-provisioner annotation already set, the PV controller might have incorrectly deleted the newly provisioned PV instead of binding it to the PVC, depending on timing and system load. ([#95909](https://github.com/kubernetes/kubernetes/pull/95909), [@pohly](https://github.com/pohly)) [SIG Apps and Storage] - -### Other (Cleanup or Flake) - -- Kubectl: the `generator` flag of `kubectl autoscale` has been deprecated and has no effect, it will be removed in a feature release ([#92998](https://github.com/kubernetes/kubernetes/pull/92998), [@SataQiu](https://github.com/SataQiu)) [SIG CLI] -- V1helpers.MatchNodeSelectorTerms now accepts just a Node and a list of Terms ([#95871](https://github.com/kubernetes/kubernetes/pull/95871), [@damemi](https://github.com/damemi)) [SIG Apps, Scheduling and Storage] -- `MatchNodeSelectorTerms` function moved to `k8s.io/component-helpers` ([#95531](https://github.com/kubernetes/kubernetes/pull/95531), [@damemi](https://github.com/damemi)) [SIG Apps, Scheduling and Storage] ## Dependencies ### Added -_Nothing has changed._ +- github.com/go-errors/errors: [v1.0.1](https://github.com/go-errors/errors/tree/v1.0.1) +- github.com/gobuffalo/here: [v0.6.0](https://github.com/gobuffalo/here/tree/v0.6.0) +- github.com/google/shlex: [e7afc7f](https://github.com/google/shlex/tree/e7afc7f) +- github.com/markbates/pkger: [v0.17.1](https://github.com/markbates/pkger/tree/v0.17.1) +- github.com/moby/spdystream: [v0.2.0](https://github.com/moby/spdystream/tree/v0.2.0) +- github.com/monochromegane/go-gitignore: [205db1a](https://github.com/monochromegane/go-gitignore/tree/205db1a) +- github.com/niemeyer/pretty: [a10e7ca](https://github.com/niemeyer/pretty/tree/a10e7ca) +- github.com/xlab/treeprint: [a009c39](https://github.com/xlab/treeprint/tree/a009c39) +- go.starlark.net: 8dd3e2e +- golang.org/x/term: 6a3ed07 +- sigs.k8s.io/kustomize/api: v0.8.5 +- sigs.k8s.io/kustomize/cmd/config: v0.9.7 +- sigs.k8s.io/kustomize/kustomize/v4: v4.0.5 +- sigs.k8s.io/kustomize/kyaml: v0.10.15 ### Changed -_Nothing has changed._ +- dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891 +- github.com/Azure/go-autorest/autorest: [v0.11.1 → v0.11.12](https://github.com/Azure/go-autorest/autorest/compare/v0.11.1...v0.11.12) +- github.com/NYTimes/gziphandler: [56545f4 → v1.1.1](https://github.com/NYTimes/gziphandler/compare/56545f4...v1.1.1) +- github.com/cilium/ebpf: [1c8d4c9 → v0.2.0](https://github.com/cilium/ebpf/compare/1c8d4c9...v0.2.0) +- github.com/container-storage-interface/spec: [v1.2.0 → v1.3.0](https://github.com/container-storage-interface/spec/compare/v1.2.0...v1.3.0) +- github.com/containerd/console: [v1.0.0 → v1.0.1](https://github.com/containerd/console/compare/v1.0.0...v1.0.1) +- github.com/containerd/containerd: [v1.4.1 → v1.4.4](https://github.com/containerd/containerd/compare/v1.4.1...v1.4.4) +- github.com/coredns/corefile-migration: [v1.0.10 → v1.0.11](https://github.com/coredns/corefile-migration/compare/v1.0.10...v1.0.11) +- github.com/creack/pty: [v1.1.7 → v1.1.11](https://github.com/creack/pty/compare/v1.1.7...v1.1.11) +- github.com/docker/docker: [bd33bbf → v20.10.2+incompatible](https://github.com/docker/docker/compare/bd33bbf...v20.10.2) +- github.com/go-logr/logr: [v0.2.0 → v0.4.0](https://github.com/go-logr/logr/compare/v0.2.0...v0.4.0) +- github.com/go-openapi/spec: [v0.19.3 → v0.19.5](https://github.com/go-openapi/spec/compare/v0.19.3...v0.19.5) +- github.com/go-openapi/strfmt: [v0.19.3 → v0.19.5](https://github.com/go-openapi/strfmt/compare/v0.19.3...v0.19.5) +- github.com/go-openapi/validate: [v0.19.5 → v0.19.8](https://github.com/go-openapi/validate/compare/v0.19.5...v0.19.8) +- github.com/gogo/protobuf: [v1.3.1 → v1.3.2](https://github.com/gogo/protobuf/compare/v1.3.1...v1.3.2) +- github.com/golang/mock: [v1.4.1 → v1.4.4](https://github.com/golang/mock/compare/v1.4.1...v1.4.4) +- github.com/google/cadvisor: [v0.38.5 → v0.39.0](https://github.com/google/cadvisor/compare/v0.38.5...v0.39.0) +- github.com/heketi/heketi: [c2e2a4a → v10.2.0+incompatible](https://github.com/heketi/heketi/compare/c2e2a4a...v10.2.0) +- github.com/kisielk/errcheck: [v1.2.0 → v1.5.0](https://github.com/kisielk/errcheck/compare/v1.2.0...v1.5.0) +- github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.2](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.2) +- github.com/kr/text: [v0.1.0 → v0.2.0](https://github.com/kr/text/compare/v0.1.0...v0.2.0) +- github.com/mattn/go-runewidth: [v0.0.2 → v0.0.7](https://github.com/mattn/go-runewidth/compare/v0.0.2...v0.0.7) +- github.com/miekg/dns: [v1.1.4 → v1.1.35](https://github.com/miekg/dns/compare/v1.1.4...v1.1.35) +- github.com/moby/sys/mountinfo: [v0.1.3 → v0.4.0](https://github.com/moby/sys/mountinfo/compare/v0.1.3...v0.4.0) +- github.com/moby/term: [672ec06 → df9cb8a](https://github.com/moby/term/compare/672ec06...df9cb8a) +- github.com/mrunalp/fileutils: [abd8a0e → v0.5.0](https://github.com/mrunalp/fileutils/compare/abd8a0e...v0.5.0) +- github.com/olekukonko/tablewriter: [a0225b3 → v0.0.4](https://github.com/olekukonko/tablewriter/compare/a0225b3...v0.0.4) +- github.com/opencontainers/runc: [v1.0.0-rc92 → v1.0.0-rc93](https://github.com/opencontainers/runc/compare/v1.0.0-rc92...v1.0.0-rc93) +- github.com/opencontainers/runtime-spec: [4d89ac9 → e6143ca](https://github.com/opencontainers/runtime-spec/compare/4d89ac9...e6143ca) +- github.com/opencontainers/selinux: [v1.6.0 → v1.8.0](https://github.com/opencontainers/selinux/compare/v1.6.0...v1.8.0) +- github.com/sergi/go-diff: [v1.0.0 → v1.1.0](https://github.com/sergi/go-diff/compare/v1.0.0...v1.1.0) +- github.com/sirupsen/logrus: [v1.6.0 → v1.7.0](https://github.com/sirupsen/logrus/compare/v1.6.0...v1.7.0) +- github.com/syndtr/gocapability: [d983527 → 42c35b4](https://github.com/syndtr/gocapability/compare/d983527...42c35b4) +- github.com/willf/bitset: [d5bec33 → v1.1.11](https://github.com/willf/bitset/compare/d5bec33...v1.1.11) +- github.com/yuin/goldmark: [v1.1.27 → v1.2.1](https://github.com/yuin/goldmark/compare/v1.1.27...v1.2.1) +- golang.org/x/crypto: 7f63de1 → 5ea612d +- golang.org/x/exp: 6cc2880 → 85be41e +- golang.org/x/mobile: d2bd2a2 → e6ae53a +- golang.org/x/mod: v0.3.0 → ce943fd +- golang.org/x/net: 69a7880 → 3d97a24 +- golang.org/x/sync: cd5d95a → 67f06af +- golang.org/x/sys: 5cba982 → a50acf3 +- golang.org/x/time: 3af7569 → f8bda1e +- golang.org/x/tools: c1934b7 → v0.1.0 +- gopkg.in/check.v1: 41f04d3 → 8fa4692 +- gopkg.in/yaml.v2: v2.2.8 → v2.4.0 +- gotest.tools/v3: v3.0.2 → v3.0.3 +- k8s.io/gengo: 83324d8 → b6c5ce2 +- k8s.io/klog/v2: v2.4.0 → v2.8.0 +- k8s.io/kube-openapi: d219536 → 591a79e +- k8s.io/system-validators: v1.2.0 → v1.4.0 +- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15 +- sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.1.0 ### Removed -_Nothing has changed._ +- github.com/codegangsta/negroni: [v1.0.0](https://github.com/codegangsta/negroni/tree/v1.0.0) +- github.com/docker/spdystream: [449fdfc](https://github.com/docker/spdystream/tree/449fdfc) +- github.com/golangplus/bytes: [45c989f](https://github.com/golangplus/bytes/tree/45c989f) +- github.com/golangplus/fmt: [2a5d6d7](https://github.com/golangplus/fmt/tree/2a5d6d7) +- github.com/gorilla/context: [v1.1.1](https://github.com/gorilla/context/tree/v1.1.1) +- github.com/kr/pty: [v1.1.5](https://github.com/kr/pty/tree/v1.1.5) +- rsc.io/quote/v3: v3.1.0 +- rsc.io/sampler: v1.3.0 +- sigs.k8s.io/kustomize: v2.0.3+incompatible -# v1.20.0-beta.0 +# v1.21.0-rc.0 -## Downloads for v1.20.0-beta.0 +## Downloads for v1.21.0-rc.0 ### Source Code filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes.tar.gz) | 385e49e32bbd6996f07bcadbf42285755b8a8ef9826ee1ba42bd82c65827cf13f63e5634b834451b263a93b708299cbb4b4b0b8ddbc688433deaf6bec240aa67 -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-src.tar.gz) | 842e80f6dcad461426fb699de8a55fde8621d76a94e54288fe9939cc1a3bbd0f4799abadac2c59bcf3f91d743726dbd17e1755312ae7fec482ef560f336dbcbb +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes.tar.gz) | ef53a41955d6f8a8d2a94636af98b55d633fb8a5081517559039e019b3dd65c9d10d4e7fa297ab88a7865d772f3eecf72e7b0eeba5e87accb4000c91da33e148 +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-src.tar.gz) | 9335a01b50d351776d3b8d00c07a5233844c51d307e361fa7e55a0620c1cb8b699e43eacf45ae9cafd8cbc44752e6987450c528a5bede8204706b7673000b5fc ### Client binaries filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-darwin-amd64.tar.gz) | bde5e7d9ee3e79d1e69465a3ddb4bb36819a4f281b5c01a7976816d7c784410812dde133cdf941c47e5434e9520701b9c5e8b94d61dca77c172f87488dfaeb26 -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-linux-386.tar.gz) | 721bb8444c9e0d7a9f8461e3f5428882d76fcb3def6eb11b8e8e08fae7f7383630699248660d69d4f6a774124d6437888666e1fa81298d5b5518bc4a6a6b2c92 -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-linux-amd64.tar.gz) | 71e4edc41afbd65f813e7ecbc22b27c95f248446f005e288d758138dc4cc708735be7218af51bcf15e8b9893a3598c45d6a685f605b46f50af3762b02c32ed76 -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-linux-arm.tar.gz) | bbefc749156f63898973f2f7c7a6f1467481329fb430d641fe659b497e64d679886482d557ebdddb95932b93de8d1e3e365c91d4bf9f110b68bd94b0ba702ded -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-linux-arm64.tar.gz) | 9803190685058b4b64d002c2fbfb313308bcea4734ed53a8c340cfdae4894d8cb13b3e819ae64051bafe0fbf8b6ecab53a6c1dcf661c57640c75b0eb60041113 -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-linux-ppc64le.tar.gz) | bcdceea64cba1ae38ea2bab50d8fd77c53f6d673de12566050b0e3c204334610e6c19e4ace763e68b5e48ab9e811521208b852b1741627be30a2b17324fc1daf -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-linux-s390x.tar.gz) | 41e36d00867e90012d5d5adfabfaae8d9f5a9fd32f290811e3c368e11822916b973afaaf43961081197f2cbab234090d97d89774e674aeadc1da61f7a64708a9 -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-windows-386.tar.gz) | c50fec5aec2d0e742f851f25c236cb73e76f8fc73b0908049a10ae736c0205b8fff83eb3d29b1748412edd942da00dd738195d9003f25b577d6af8359d84fb2f -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-client-windows-amd64.tar.gz) | 0fd6777c349908b6d627e849ea2d34c048b8de41f7df8a19898623f597e6debd35b7bcbf8e1d43a1be3a9abb45e4810bc498a0963cf780b109e93211659e9c7e +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | 964135e43234cee275c452f5f06fb6d2bcd3cff3211a0d50fa35fff1cc4446bc5a0ac5125405dadcfb6596cb152afe29fabf7aad5b35b100e1288db890b70f8e +[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-darwin-arm64.tar.gz) | 50d782abaa4ded5e706b3192d87effa953ceabbd7d91e3d48b0c1fa2206a1963a909c14b923560f5d09cac2c7392edc5f38a13fbf1e9a40bc94e3afe8de10622 +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-386.tar.gz) | 72af5562f24184a2d7c27f95fa260470da979fbdcacce39a372f8f3add2991d7af8bc78f4e1dbe7a0f97e3f559b149b72a51491d3b13008da81872ee50f02f37 +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 1eddb8f6b51e005bc6f7b519d036cbe3d2f6d97dbf7d212dd933fb56354c29f222d050519115a9bcf94555aef095db7cf763469e47bb4ae3c6c07f97edf437cb +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-arm.tar.gz) | 670f8ca60ea3cf0bb3262a772715e0ea735fccda6a92f3186299361dc455b304ae177d4017e0b67bbfa4a95e36f4cc3f7eb335e2a5130c93ac3fba2aff4519bf +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | a69a47907cff138ba393d8c87044fd95d97f3ca8f35d301b50742e2801ad7c229d99d6667971091f65825eb51854d585be0dd7421670110b1aa567e67e7ab4b3 +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | b929feade94b71c81908abdcd4343b1e1e20098fd65e10d4d02585ad649d292d06f52c7ddc349efa188ce5b093e703c7aa9582c6ae5a69699adb87bbf5350243 +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 899d1470e412282cf289d8e24806d1a08c62ec0151f345ae3c9e497cc7bc0feab76498de4dd897d6adcdfa0c422e6b1a37e25d928669030f53457fd69d6e7df7 +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-windows-386.tar.gz) | 9f0bc90a269eabd06fe4f637b5172a3a6a7d3de26de0d66504c2e1f2093083c584ea39031db6075a7da7a86b98c48bed25aa88d4ac09060b38692c6a5b637078 +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | 05c8cc10188a1294b0d51d052942742a9b26411a08ec73494bf0e728a8a167e0a7863bdfc8864e76a371b584380098381805341e18b4b283b5d0cf298d5f7c7c ### Server binaries filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-server-linux-amd64.tar.gz) | 30d982424ca64bf0923503ae8195b2e2a59497096b2d9e58dfd491cd6639633027acfa9750bc7bccf34e1dc116d29d2f87cbd7ae713db4210ce9ac16182f0576 -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-server-linux-arm.tar.gz) | f08b62be9bc6f0745f820b0083c7a31eedb2ce370a037c768459a59192107b944c8f4345d0bb88fc975f2e7a803ac692c9ac3e16d4a659249d4600e84ff75d9e -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-server-linux-arm64.tar.gz) | e3472b5b3dfae0a56e5363d52062b1e4a9fc227a05e0cf5ece38233b2c442f427970aab94a52377fb87e583663c120760d154bc1c4ac22dca1f4d0d1ebb96088 -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-server-linux-ppc64le.tar.gz) | 06c254e0a62f755d31bc40093d86c44974f0a60308716cc3214a6b3c249a4d74534d909b82f8a3dd3a3c9720e61465b45d2bb3a327ef85d3caba865750020dfb -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-server-linux-s390x.tar.gz) | 2edeb4411c26a0de057a66787091ab1044f71774a464aed898ffee26634a40127181c2edddb38e786b6757cca878fd0c3a885880eec6c3448b93c645770abb12 +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | 355f278728ef7ac7eb2f5568c99c1429543c6302bbd0ed3bd0378c08116075e56ae850a49241313f078e2392702672ec6c9b70c8d97b4f2f5f4bee36828a63ba +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-arm.tar.gz) | 9ac02c2825e2fd4e92f0c0f67180c67c24e32841ccbabc82284bf6293727ffecfae65e8a42b527c2a7ca482752384928eb65c2a1706144ae7819a6b3a1ab291c +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | eb412453da03c82a9248412c8ccf4d4baa1fbfa81edd8d4f81d28969b40a3727e18934accc68f643d253446c58ffd2623292402495480b3d4b2a837b5318b957 +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | 07da2812c35bbc427ee5b4a0b601c3ae271e0d50ab0dd4c5c25399f43506fa2a187642eb9d4d2085df7b90264d48ea2f31088af87d9efa7eb2e87f91e1fdbde4 +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 3b79442a3d6e389c4ff105922a8e49994c0b6c088d2c501bd8c78d9f9e814902f5bb72c8f9c89380b750fda9b3a336759b9b68f11d70bef4f0e984564a95c29e ### Node binaries filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-node-linux-amd64.tar.gz) | cc1d5b94b86070b5e7746d7aaeaeac3b3a5e5ebbff1ec33885f7eeab270a6177d593cb1975b2e56f4430b7859ad42da76f266629f9313e0f688571691ac448ed -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-node-linux-arm.tar.gz) | 75e82c7c9122add3b24695b94dcb0723c52420c3956abf47511e37785aa48a1fa8257db090c6601010c4475a325ccfff13eb3352b65e3aa1774f104b09b766b0 -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-node-linux-arm64.tar.gz) | 16ef27c40bf4d678a55fcd3d3f7d09f1597eec2cc58f9950946f0901e52b82287be397ad7f65e8d162d8a9cdb4a34a610b6db8b5d0462be8e27c4b6eb5d6e5e7 -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-node-linux-ppc64le.tar.gz) | 939865f2c4cb6a8934f22a06223e416dec5f768ffc1010314586149470420a1d62aef97527c34d8a636621c9669d6489908ce1caf96f109e8d073cee1c030b50 -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-node-linux-s390x.tar.gz) | bbfdd844075fb816079af7b73d99bc1a78f41717cdbadb043f6f5872b4dc47bc619f7f95e2680d4b516146db492c630c17424e36879edb45e40c91bc2ae4493c -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-beta.0/kubernetes-node-windows-amd64.tar.gz) | a2b3ea40086fd71aed71a4858fd3fc79fd1907bc9ea8048ff3c82ec56477b0a791b724e5a52d79b3b36338c7fbd93dfd3d03b00ccea9042bda0d270fc891e4ec +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | f12edf1faf5f07de1ebc5a8626601c12927902e10aca3f11e398637382fdf55365dbd9a0ef38858553fb7569495ae2cf68f155dd2e49b85b27d76fb599bb92e4 +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-arm.tar.gz) | 4fba8fc4e2102f07fb778aab597ec7231ea65c35e1aa618fe98b707b64a931237bd842c173e9120326e4d9deb983bb3917176762bba2212612bbc09d6e2105c4 +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | a2e1be5459a8346839970faf4e7ebdb8ab9f3273e02babf1f3199b06bdb67434a2d18fcd1628cf1b989756e99d8dad6624a455b9db11d50f51f509f4df5c27da +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 16d2c1cc295474fc49fe9a827ddd73e81bdd6b76af7074987b90250023f99b6d70bf474e204c7d556802111984fcb3a330740b150bdc7970d0e3634eb94a1665 +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | 9dc6faa6cd007b13dfce703f3e271f80adcc4e029c90a4a9b4f2f143b9756f2893f8af3d7c2cf813f2bd6731cffd87d15d4229456c1685939f65bf467820ec6e +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | f8bac2974c9142bfb80cd5eadeda79f79f27b78899a4e6e71809b795c708824ba442be83fdbadb98e01c3823dd8350776358258a205e851ed045572923cacba7 -## Changelog since v1.20.0-alpha.3 +## Changelog since v1.21.0-beta.1 ## Urgent Upgrade Notes ### (No, really, you MUST read this before you upgrade) - - Kubeadm: improve the validation of serviceSubnet and podSubnet. - ServiceSubnet has to be limited in size, due to implementation details, and the mask can not allocate more than 20 bits. - PodSubnet validates against the corresponding cluster "--node-cidr-mask-size" of the kube-controller-manager, it fail if the values are not compatible. - kubeadm no longer sets the node-mask automatically on IPv6 deployments, you must check that your IPv6 service subnet mask is compatible with the default node mask /64 or set it accordenly. - Previously, for IPv6, if the podSubnet had a mask lower than /112, kubeadm calculated a node-mask to be multiple of eight and splitting the available bits to maximise the number used for nodes. ([#95723](https://github.com/kubernetes/kubernetes/pull/95723), [@aojea](https://github.com/aojea)) [SIG Cluster Lifecycle] - - Windows hyper-v container featuregate is deprecated in 1.20 and will be removed in 1.21 ([#95505](https://github.com/kubernetes/kubernetes/pull/95505), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] + - Migrated pkg/kubelet/cm/cpuset/cpuset.go to structured logging. Exit code changed from 255 to 1. ([#100007](https://github.com/kubernetes/kubernetes/pull/100007), [@utsavoza](https://github.com/utsavoza)) [SIG Instrumentation and Node] ## Changes by Kind -### Deprecation - -- Support 'controlplane' as a valid EgressSelection type in the EgressSelectorConfiguration API. 'Master' is deprecated and will be removed in v1.22. ([#95235](https://github.com/kubernetes/kubernetes/pull/95235), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] - ### API Change -- Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. - It changes the dual-stack API wrt Service from a single ipFamily field to 3 - fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), - ipFamilies (a list of families assigned), and clusterIPs (inclusive of - clusterIP). Most users do not need to set anything at all, defaulting will - handle it for them. Services are single-stack unless the user asks for - dual-stack. This is all gated by the "IPv6DualStack" feature gate. ([#91824](https://github.com/kubernetes/kubernetes/pull/91824), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] -- Introduces a metric source for HPAs which allows scaling based on container resource usage. ([#90691](https://github.com/kubernetes/kubernetes/pull/90691), [@arjunrn](https://github.com/arjunrn)) [SIG API Machinery, Apps, Autoscaling and CLI] +- Add Probe-level terminationGracePeriodSeconds field ([#99375](https://github.com/kubernetes/kubernetes/pull/99375), [@ehashman](https://github.com/ehashman)) [SIG API Machinery, Apps, Node and Testing] +- CSIServiceAccountToken is Beta now ([#99298](https://github.com/kubernetes/kubernetes/pull/99298), [@zshihang](https://github.com/zshihang)) [SIG Auth, Storage and Testing] +- Discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. ([#100472](https://github.com/kubernetes/kubernetes/pull/100472), [@liggitt](https://github.com/liggitt)) [SIG Network] +- FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). ([#99661](https://github.com/kubernetes/kubernetes/pull/99661), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery, Auth and Testing] +- Generic ephemeral volumes are beta. ([#99643](https://github.com/kubernetes/kubernetes/pull/99643), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing] +- Implement the GetAvailableResources in the podresources API. ([#95734](https://github.com/kubernetes/kubernetes/pull/95734), [@fromanirh](https://github.com/fromanirh)) [SIG Instrumentation, Node and Testing] +- The Endpoints controller will now set the `endpoints.kubernetes.io/over-capacity` annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. ([#99975](https://github.com/kubernetes/kubernetes/pull/99975), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector (`{}`) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. ([#99290](https://github.com/kubernetes/kubernetes/pull/99290), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing] +- Topology Aware Hints are now available in alpha and can be enabled with the `TopologyAwareHints` feature gate. ([#99522](https://github.com/kubernetes/kubernetes/pull/99522), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing] ### Feature -- Add a metric for time taken to perform recursive permission change ([#95866](https://github.com/kubernetes/kubernetes/pull/95866), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] -- Allow cross compilation of kubernetes on different platforms. ([#94403](https://github.com/kubernetes/kubernetes/pull/94403), [@bnrjee](https://github.com/bnrjee)) [SIG Release] -- Command to start network proxy changes from 'KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh' to 'KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh' ([#92669](https://github.com/kubernetes/kubernetes/pull/92669), [@Jefftree](https://github.com/Jefftree)) [SIG Cloud Provider] -- DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. ([#95631](https://github.com/kubernetes/kubernetes/pull/95631), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling and Testing] -- Kubernetes E2E test image manifest lists now contain Windows images. ([#77398](https://github.com/kubernetes/kubernetes/pull/77398), [@claudiubelu](https://github.com/claudiubelu)) [SIG Testing and Windows] -- Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added the pause:3.4 image. ([#91452](https://github.com/kubernetes/kubernetes/pull/91452), [@claudiubelu](https://github.com/claudiubelu)) [SIG Node, Release and Windows] - -### Documentation - -- Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList ([#95117](https://github.com/kubernetes/kubernetes/pull/95117), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] +- Add e2e test to validate performance metrics of volume lifecycle operations ([#94334](https://github.com/kubernetes/kubernetes/pull/94334), [@RaunakShah](https://github.com/RaunakShah)) [SIG Storage and Testing] +- EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. ([#100319](https://github.com/kubernetes/kubernetes/pull/100319), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] +- Enables Kubelet to check volume condition and log events to corresponding pods. ([#99284](https://github.com/kubernetes/kubernetes/pull/99284), [@fengzixu](https://github.com/fengzixu)) [SIG Apps, Instrumentation, Node and Storage] +- Introduce a churn operator to scheduler perf testing framework. ([#98900](https://github.com/kubernetes/kubernetes/pull/98900), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] +- Kubernetes is now built with Golang 1.16.1 ([#100106](https://github.com/kubernetes/kubernetes/pull/100106), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] +- Migrated pkg/kubelet/cm/devicemanager to structured logging ([#99976](https://github.com/kubernetes/kubernetes/pull/99976), [@knabben](https://github.com/knabben)) [SIG Instrumentation and Node] +- Migrated pkg/kubelet/cm/memorymanager to structured logging ([#99974](https://github.com/kubernetes/kubernetes/pull/99974), [@knabben](https://github.com/knabben)) [SIG Instrumentation and Node] +- Migrated pkg/kubelet/cm/topologymanager to structure logging ([#99969](https://github.com/kubernetes/kubernetes/pull/99969), [@knabben](https://github.com/knabben)) [SIG Instrumentation and Node] +- Rename metrics `etcd_object_counts` to `apiserver_storage_object_counts` and mark it as stable. The original `etcd_object_counts` metrics name is marked as "Deprecated" and will be removed in the future. ([#99785](https://github.com/kubernetes/kubernetes/pull/99785), [@erain](https://github.com/erain)) [SIG API Machinery, Instrumentation and Testing] +- Update pause container to run as pseudo user and group `65535:65535`. This implies the release of version 3.5 of the container images. ([#97963](https://github.com/kubernetes/kubernetes/pull/97963), [@saschagrunert](https://github.com/saschagrunert)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing] +- Users might specify the `kubectl.kubernetes.io/default-exec-container` annotation in a Pod to preselect container for kubectl commands. ([#99833](https://github.com/kubernetes/kubernetes/pull/99833), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG CLI] ### Bug or Regression -- Exposes and sets a default timeout for the SubjectAccessReview client for DelegatingAuthorizationOptions. ([#95725](https://github.com/kubernetes/kubernetes/pull/95725), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Cloud Provider] -- Alter wording to describe pods using a pvc ([#95635](https://github.com/kubernetes/kubernetes/pull/95635), [@RaunakShah](https://github.com/RaunakShah)) [SIG CLI] -- If we set SelectPolicy MinPolicySelect on scaleUp behavior or scaleDown behavior,Horizontal Pod Autoscaler doesn`t automatically scale the number of pods correctly ([#95647](https://github.com/kubernetes/kubernetes/pull/95647), [@JoshuaAndrew](https://github.com/JoshuaAndrew)) [SIG Apps and Autoscaling] -- Ignore apparmor for non-linux operating systems ([#93220](https://github.com/kubernetes/kubernetes/pull/93220), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] -- Ipvs: ensure selected scheduler kernel modules are loaded ([#93040](https://github.com/kubernetes/kubernetes/pull/93040), [@cmluciano](https://github.com/cmluciano)) [SIG Network] -- Kubeadm: add missing "--experimental-patches" flag to "kubeadm init phase control-plane" ([#95786](https://github.com/kubernetes/kubernetes/pull/95786), [@Sh4d1](https://github.com/Sh4d1)) [SIG Cluster Lifecycle] -- Reorganized iptables rules to fix a performance issue ([#95252](https://github.com/kubernetes/kubernetes/pull/95252), [@tssurya](https://github.com/tssurya)) [SIG Network] -- Unhealthy pods covered by PDBs can be successfully evicted if enough healthy pods are available. ([#94381](https://github.com/kubernetes/kubernetes/pull/94381), [@michaelgugino](https://github.com/michaelgugino)) [SIG Apps] -- Update the PIP when it is not in the Succeeded provisioning state during the LB update. ([#95748](https://github.com/kubernetes/kubernetes/pull/95748), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Update the frontend IP config when the service's `pipName` annotation is changed ([#95813](https://github.com/kubernetes/kubernetes/pull/95813), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Add ability to skip OpenAPI handler installation to the GenericAPIServer ([#100341](https://github.com/kubernetes/kubernetes/pull/100341), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery] +- Count pod overhead against an entity's ResourceQuota ([#99600](https://github.com/kubernetes/kubernetes/pull/99600), [@gjkim42](https://github.com/gjkim42)) [SIG API Machinery and Node] +- EndpointSlice controllers are less likely to create duplicate EndpointSlices. ([#100103](https://github.com/kubernetes/kubernetes/pull/100103), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- Ensure only one LoadBalancer rule is created when HA mode is enabled ([#99825](https://github.com/kubernetes/kubernetes/pull/99825), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] +- Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. ([#95783](https://github.com/kubernetes/kubernetes/pull/95783), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] +- Fixed authentication_duration_seconds metric. Previously it included whole apiserver request duration. ([#99944](https://github.com/kubernetes/kubernetes/pull/99944), [@marseel](https://github.com/marseel)) [SIG API Machinery, Instrumentation and Scalability] +- Fixes issue where inline AzueFile secrets could not be accessed from the pod's namespace. ([#100563](https://github.com/kubernetes/kubernetes/pull/100563), [@msau42](https://github.com/msau42)) [SIG Storage] +- Improve speed of vSphere PV provisioning and reduce number of API calls ([#100054](https://github.com/kubernetes/kubernetes/pull/100054), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] +- Kubectl: Fixed panic when describing an ingress backend without an API Group ([#100505](https://github.com/kubernetes/kubernetes/pull/100505), [@lauchokyip](https://github.com/lauchokyip)) [SIG CLI] +- Kubectl: fix case of age column in describe node (#96963, @bl-ue) ([#96963](https://github.com/kubernetes/kubernetes/pull/96963), [@bl-ue](https://github.com/bl-ue)) [SIG CLI] +- Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. ([#96616](https://github.com/kubernetes/kubernetes/pull/96616), [@perithompson](https://github.com/perithompson)) [SIG Node and Windows] +- Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows ([#97378](https://github.com/kubernetes/kubernetes/pull/97378), [@hwdef](https://github.com/hwdef)) [SIG Node and Windows] +- Pass `KUBE_BUILD_CONFORMANCE=y` to the package-tarballs to reenable building the conformance tarballs. ([#100571](https://github.com/kubernetes/kubernetes/pull/100571), [@puerco](https://github.com/puerco)) [SIG Release] +- Pod Log stats for windows now reports metrics ([#99221](https://github.com/kubernetes/kubernetes/pull/99221), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node, Storage, Testing and Windows] ### Other (Cleanup or Flake) -- NO ([#95690](https://github.com/kubernetes/kubernetes/pull/95690), [@nikhita](https://github.com/nikhita)) [SIG Release] - -## Dependencies - -### Added -- github.com/form3tech-oss/jwt-go: [v3.2.2+incompatible](https://github.com/form3tech-oss/jwt-go/tree/v3.2.2) - -### Changed -- github.com/Azure/go-autorest/autorest/adal: [v0.9.0 → v0.9.5](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.9.0...v0.9.5) -- github.com/Azure/go-autorest/autorest/mocks: [v0.4.0 → v0.4.1](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.4.0...v0.4.1) -- golang.org/x/crypto: 75b2880 → 7f63de1 - -### Removed -_Nothing has changed._ - - - -# v1.20.0-alpha.3 - - -## Downloads for v1.20.0-alpha.3 - -### Source Code - -filename | sha512 hash --------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes.tar.gz) | 542cc9e0cd97732020491456402b6e2b4f54f2714007ee1374a7d363663a1b41e82b50886176a5313aaccfbfd4df2bc611d6b32d19961cdc98b5821b75d6b17c -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-src.tar.gz) | 5e5d725294e552fd1d14fd6716d013222827ac2d4e2d11a7a1fdefb77b3459bbeb69931f38e1597de205dd32a1c9763ab524c2af1551faef4f502ef0890f7fbf - -### Client binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | 60004939727c75d0f06adc4449e16b43303941937c0e9ea9aca7d947e93a5aed5d11e53d1fc94caeb988be66d39acab118d406dc2d6cead61181e1ced6d2be1a -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-linux-386.tar.gz) | 7edba9c4f1bf38fdf1fa5bff2856c05c0e127333ce19b17edf3119dc9b80462c027404a1f58a5eabf1de73a8f2f20aced043dda1fafd893619db1a188cda550c -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | db1818aa82d072cb3e32a2a988e66d76ecf7cebc6b8a29845fa2d6ec27f14a36e4b9839b1b7ed8c43d2da9cde00215eb672a7e8ee235d2e3107bc93c22e58d38 -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | d2922e70d22364b1f5a1e94a0c115f849fe2575b231b1ba268f73a9d86fc0a9fbb78dc713446839a2593acf1341cb5a115992f350870f13c1a472bb107b75af7 -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | 2e3ae20e554c7d4fc3a8afdfcafe6bbc81d4c5e9aea036357baac7a3fdc2e8098aa8a8c3dded3951667d57f667ce3fbf37ec5ae5ceb2009a569dc9002d3a92f9 -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | b54a34e572e6a86221577de376e6f7f9fcd82327f7fe94f2fc8d21f35d302db8a0f3d51e60dc89693999f5df37c96d0c3649a29f07f095efcdd59923ae285c95 -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | 5be1b70dc437d3ba88cb0b89cd1bc555f79896c3f5b5f4fa0fb046a0d09d758b994d622ebe5cef8e65bba938c5ae945b81dc297f9dfa0d98f82ea75f344a3a0d -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-windows-386.tar.gz) | 88cf3f66168ef3bf9a5d3d2275b7f33799406e8205f2c202997ebec23d449aa4bb48b010356ab1cf52ff7b527b8df7c8b9947a43a82ebe060df83c3d21b7223a -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | 87d2d4ea1829da8cfa1a705a03ea26c759a03bd1c4d8b96f2c93264c4d172bb63a91d9ddda65cdc5478b627c30ae8993db5baf8be262c157d83bffcebe85474e - -### Server binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | 7af691fc0b13a937797912374e3b3eeb88d5262e4eb7d4ebe92a3b64b3c226cb049aedfd7e39f639f6990444f7bcf2fe58699cf0c29039daebe100d7eebf60de -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-server-linux-arm.tar.gz) | 557c47870ecf5c2090b2694c8f0c8e3b4ca23df5455a37945bd037bc6fb5b8f417bf737bb66e6336b285112cb52de0345240fdb2f3ce1c4fb335ca7ef1197f99 -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | 981de6cf7679d743cdeef1e894314357b68090133814801870504ef30564e32b5675e270db20961e9a731e35241ad9b037bdaf749da87b6c4ce8889eeb1c5855 -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | 506578a21601ccff609ae757a55e68634c15cbfecbf13de972c96b32a155ded29bd71aee069c77f5f721416672c7a7ac0b8274de22bfd28e1ecae306313d96c5 -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | af0cdcd4a77a7cc8060a076641615730a802f1f02dab084e41926023489efec6102d37681c70ab0dbe7440cd3e72ea0443719a365467985360152b9aae657375 - -### Node binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | 2d92c61596296279de1efae23b2b707415565d9d50cd61a7231b8d10325732b059bcb90f3afb36bef2575d203938c265572721e38df408e8792d3949523bd5d9 -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-node-linux-arm.tar.gz) | c298de9b5ac1b8778729a2d8e2793ff86743033254fbc27014333880b03c519de81691caf03aa418c729297ee8942ce9ec89d11b0e34a80576b9936015dc1519 -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | daa3c65afda6d7aff206c1494390bbcc205c2c6f8db04c10ca967a690578a01c49d49c6902b85e7158f79fd4d2a87c5d397d56524a75991c9d7db85ac53059a7 -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | 05661908bb73bfcaf9c2eae96e9a6a793db5a7a100bce6df9e057985dd53a7a5248d72e81b6d13496bd38b9326c17cdb2edaf0e982b6437507245fb846e1efc6 -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | 845e518e2c4ef0cef2c3b58f0b9ea5b5fe9b8a249717f789607752484c424c26ae854b263b7c0a004a8426feb9aa3683c177a9ed2567e6c3521f4835ea08c24a -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | 530e536574ed2c3e5973d3c0f0fdd2b4d48ef681a7a7c02db13e605001669eeb4f4b8a856fc08fc21436658c27b377f5d04dbcb3aae438098abc953b6eaf5712 - -## Changelog since v1.20.0-alpha.2 - -## Changes by Kind - -### API Change - -- New parameter `defaultingType` for `PodTopologySpread` plugin allows to use k8s defined or user provided default constraints ([#95048](https://github.com/kubernetes/kubernetes/pull/95048), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - -### Feature - -- Added new k8s.io/component-helpers repository providing shared helper code for (core) components. ([#92507](https://github.com/kubernetes/kubernetes/pull/92507), [@ingvagabund](https://github.com/ingvagabund)) [SIG Apps, Node, Release and Scheduling] -- Adds `create ingress` command to `kubectl` ([#78153](https://github.com/kubernetes/kubernetes/pull/78153), [@amimof](https://github.com/amimof)) [SIG CLI and Network] -- Kubectl create now supports creating ingress objects. ([#94327](https://github.com/kubernetes/kubernetes/pull/94327), [@rikatz](https://github.com/rikatz)) [SIG CLI and Network] -- New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used ([#95539](https://github.com/kubernetes/kubernetes/pull/95539), [@soulxu](https://github.com/soulxu)) [SIG Scheduling] -- SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. - Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, - and note that some cloud platforms and network plugins do not support SCTP traffic. ([#95566](https://github.com/kubernetes/kubernetes/pull/95566), [@danwinship](https://github.com/danwinship)) [SIG Apps and Network] -- Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. ([#93534](https://github.com/kubernetes/kubernetes/pull/93534), [@everpeace](https://github.com/everpeace)) [SIG Scheduling and Testing] -- SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled ([#95448](https://github.com/kubernetes/kubernetes/pull/95448), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] -- SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. ([#95267](https://github.com/kubernetes/kubernetes/pull/95267), [@javidiaz](https://github.com/javidiaz)) [SIG Node] - -### Bug or Regression - -- An issues preventing volume expand controller to annotate the PVC with `volume.kubernetes.io/storage-resizer` when the PVC StorageClass is already updated to the out-of-tree provisioner is now fixed. ([#94489](https://github.com/kubernetes/kubernetes/pull/94489), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG API Machinery, Apps and Storage] -- Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. ([#94916](https://github.com/kubernetes/kubernetes/pull/94916), [@smileusd](https://github.com/smileusd)) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing] -- Fix azure disk attach failure for disk size bigger than 4TB ([#95463](https://github.com/kubernetes/kubernetes/pull/95463), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix azure disk data loss issue on Windows when unmount disk ([#95456](https://github.com/kubernetes/kubernetes/pull/95456), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Fix verb & scope reporting for kube-apiserver metrics (LIST reported instead of GET) ([#95562](https://github.com/kubernetes/kubernetes/pull/95562), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] -- Fix vsphere detach failure for static PVs ([#95447](https://github.com/kubernetes/kubernetes/pull/95447), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] -- Fix: smb valid path error ([#95583](https://github.com/kubernetes/kubernetes/pull/95583), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] -- Fixed a bug causing incorrect formatting of `kubectl describe ingress`. ([#94985](https://github.com/kubernetes/kubernetes/pull/94985), [@howardjohn](https://github.com/howardjohn)) [SIG CLI and Network] -- Fixed a bug in client-go where new clients with customized `Dial`, `Proxy`, `GetCert` config may get stale HTTP transports. ([#95427](https://github.com/kubernetes/kubernetes/pull/95427), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] -- Fixes high CPU usage in kubectl drain ([#95260](https://github.com/kubernetes/kubernetes/pull/95260), [@amandahla](https://github.com/amandahla)) [SIG CLI] -- Support the node label `node.kubernetes.io/exclude-from-external-load-balancers` ([#95542](https://github.com/kubernetes/kubernetes/pull/95542), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - -### Other (Cleanup or Flake) - -- Fix func name NewCreateCreateDeploymentOptions ([#91931](https://github.com/kubernetes/kubernetes/pull/91931), [@lixiaobing1](https://github.com/lixiaobing1)) [SIG CLI] -- Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 ([#95419](https://github.com/kubernetes/kubernetes/pull/95419), [@jsturtevant](https://github.com/jsturtevant)) [SIG Cluster Lifecycle and Windows] -- Upgrade snapshot controller to 3.0.0 ([#95412](https://github.com/kubernetes/kubernetes/pull/95412), [@saikat-royc](https://github.com/saikat-royc)) [SIG Cloud Provider] -- Remove the dependency of csi-translation-lib module on apiserver/cloud-provider/controller-manager ([#95543](https://github.com/kubernetes/kubernetes/pull/95543), [@wawa0210](https://github.com/wawa0210)) [SIG Release] -- Scheduler framework interface moved from pkg/scheduler/framework/v1alpha to pkg/scheduler/framework ([#95069](https://github.com/kubernetes/kubernetes/pull/95069), [@farah](https://github.com/farah)) [SIG Scheduling, Storage and Testing] -- UDP and SCTP protocols can left stale connections that need to be cleared to avoid services disruption, but they can cause problems that are hard to debug. - Kubernetes components using a loglevel greater or equal than 4 will log the conntrack operations and its output, to show the entries that were deleted. ([#95694](https://github.com/kubernetes/kubernetes/pull/95694), [@aojea](https://github.com/aojea)) [SIG Network] +- A new storage E2E testsuite covers CSIStorageCapacity publishing if a driver opts into the test. ([#100537](https://github.com/kubernetes/kubernetes/pull/100537), [@pohly](https://github.com/pohly)) [SIG Storage and Testing] +- Convert cmd/kubelet/app/server.go to structured logging ([#98334](https://github.com/kubernetes/kubernetes/pull/98334), [@wawa0210](https://github.com/wawa0210)) [SIG Node] +- If kube-apiserver enabled goaway feature, clients required golang 1.15.8 or 1.16+ version to avoid un-expected data race issue. ([#98809](https://github.com/kubernetes/kubernetes/pull/98809), [@answer1991](https://github.com/answer1991)) [SIG API Machinery] +- Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. ([#98753](https://github.com/kubernetes/kubernetes/pull/98753), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps and Storage] +- Migrate `pkg/kubelet/pluginmanager` to structured logging ([#99885](https://github.com/kubernetes/kubernetes/pull/99885), [@qingwave](https://github.com/qingwave)) [SIG Node] +- Migrate `pkg/kubelet/preemption/preemption.go` and `pkg/kubelet/logs/container_log_manager.go` to structured logging ([#99848](https://github.com/kubernetes/kubernetes/pull/99848), [@qingwave](https://github.com/qingwave)) [SIG Node] +- Migrate `pkg/kubelet/(cri)` to structured logging ([#99006](https://github.com/kubernetes/kubernetes/pull/99006), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] +- Migrate `pkg/kubelet/(node, pod)` to structured logging ([#98847](https://github.com/kubernetes/kubernetes/pull/98847), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] +- Migrate `pkg/kubelet/(volume,container)` to structured logging ([#98850](https://github.com/kubernetes/kubernetes/pull/98850), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] +- Migrate `pkg/kubelet/kubelet_node_status.go` to structured logging ([#98154](https://github.com/kubernetes/kubernetes/pull/98154), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node and Release] +- Migrate `pkg/kubelet/lifecycle,oom` to structured logging ([#99479](https://github.com/kubernetes/kubernetes/pull/99479), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Node] +- Migrate cmd/kubelet/+ pkg/kubelet/cadvisor/cadvisor_linux.go + pkg/kubelet/cri/remote/util/util_unix.go + pkg/kubelet/images/image_manager.go to structured logging ([#99994](https://github.com/kubernetes/kubernetes/pull/99994), [@AfrouzMashayekhi](https://github.com/AfrouzMashayekhi)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/cm/container_manager_linux.go and pkg/kubelet/cm/container_manager_stub.go to structured logging ([#100001](https://github.com/kubernetes/kubernetes/pull/100001), [@shiyajuan123](https://github.com/shiyajuan123)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/cm/cpumanage/{topology/togit pology.go, policy_none.go, cpu_assignment.go} to structured logging ([#100163](https://github.com/kubernetes/kubernetes/pull/100163), [@lala123912](https://github.com/lala123912)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/cm/cpumanager/state to structured logging ([#99563](https://github.com/kubernetes/kubernetes/pull/99563), [@jmguzik](https://github.com/jmguzik)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/config to structured logging ([#100002](https://github.com/kubernetes/kubernetes/pull/100002), [@AfrouzMashayekhi](https://github.com/AfrouzMashayekhi)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/kubelet.go to structured logging ([#99861](https://github.com/kubernetes/kubernetes/pull/99861), [@navidshaikh](https://github.com/navidshaikh)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/kubeletconfig to structured logging ([#100265](https://github.com/kubernetes/kubernetes/pull/100265), [@ehashman](https://github.com/ehashman)) [SIG Node] +- Migrate pkg/kubelet/kuberuntime to structured logging ([#99970](https://github.com/kubernetes/kubernetes/pull/99970), [@krzysiekg](https://github.com/krzysiekg)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/prober to structured logging ([#99830](https://github.com/kubernetes/kubernetes/pull/99830), [@krzysiekg](https://github.com/krzysiekg)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/winstats to structured logging ([#99855](https://github.com/kubernetes/kubernetes/pull/99855), [@hexxdump](https://github.com/hexxdump)) [SIG Instrumentation and Node] +- Migrate probe log messages to structured logging ([#97093](https://github.com/kubernetes/kubernetes/pull/97093), [@aldudko](https://github.com/aldudko)) [SIG Instrumentation and Node] +- Migrate remaining kubelet files to structured logging ([#100196](https://github.com/kubernetes/kubernetes/pull/100196), [@ehashman](https://github.com/ehashman)) [SIG Instrumentation and Node] +- `apiserver_storage_objects` (a newer version of `etcd_object_counts) is promoted and marked as stable. ([#100082](https://github.com/kubernetes/kubernetes/pull/100082), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] ## Dependencies @@ -1606,536 +774,942 @@ filename | sha512 hash _Nothing has changed._ ### Changed -_Nothing has changed._ +- github.com/cilium/ebpf: [1c8d4c9 → v0.2.0](https://github.com/cilium/ebpf/compare/1c8d4c9...v0.2.0) +- github.com/containerd/console: [v1.0.0 → v1.0.1](https://github.com/containerd/console/compare/v1.0.0...v1.0.1) +- github.com/containerd/containerd: [v1.4.1 → v1.4.4](https://github.com/containerd/containerd/compare/v1.4.1...v1.4.4) +- github.com/creack/pty: [v1.1.9 → v1.1.11](https://github.com/creack/pty/compare/v1.1.9...v1.1.11) +- github.com/docker/docker: [bd33bbf → v20.10.2+incompatible](https://github.com/docker/docker/compare/bd33bbf...v20.10.2) +- github.com/google/cadvisor: [v0.38.8 → v0.39.0](https://github.com/google/cadvisor/compare/v0.38.8...v0.39.0) +- github.com/konsorten/go-windows-terminal-sequences: [v1.0.3 → v1.0.2](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.3...v1.0.2) +- github.com/moby/sys/mountinfo: [v0.1.3 → v0.4.0](https://github.com/moby/sys/mountinfo/compare/v0.1.3...v0.4.0) +- github.com/moby/term: [672ec06 → df9cb8a](https://github.com/moby/term/compare/672ec06...df9cb8a) +- github.com/mrunalp/fileutils: [abd8a0e → v0.5.0](https://github.com/mrunalp/fileutils/compare/abd8a0e...v0.5.0) +- github.com/opencontainers/runc: [v1.0.0-rc92 → v1.0.0-rc93](https://github.com/opencontainers/runc/compare/v1.0.0-rc92...v1.0.0-rc93) +- github.com/opencontainers/runtime-spec: [4d89ac9 → e6143ca](https://github.com/opencontainers/runtime-spec/compare/4d89ac9...e6143ca) +- github.com/opencontainers/selinux: [v1.6.0 → v1.8.0](https://github.com/opencontainers/selinux/compare/v1.6.0...v1.8.0) +- github.com/sirupsen/logrus: [v1.6.0 → v1.7.0](https://github.com/sirupsen/logrus/compare/v1.6.0...v1.7.0) +- github.com/syndtr/gocapability: [d983527 → 42c35b4](https://github.com/syndtr/gocapability/compare/d983527...42c35b4) +- github.com/willf/bitset: [d5bec33 → v1.1.11](https://github.com/willf/bitset/compare/d5bec33...v1.1.11) +- gotest.tools/v3: v3.0.2 → v3.0.3 +- k8s.io/klog/v2: v2.5.0 → v2.8.0 +- sigs.k8s.io/structured-merge-diff/v4: v4.0.3 → v4.1.0 ### Removed _Nothing has changed._ -# v1.20.0-alpha.2 +# v1.21.0-beta.1 -## Downloads for v1.20.0-alpha.2 +## Downloads for v1.21.0-beta.1 ### Source Code filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes.tar.gz) | 45089a4d26d56a5d613ecbea64e356869ac738eca3cc71d16b74ea8ae1b4527bcc32f1dc35ff7aa8927e138083c7936603faf063121d965a2f0f8ba28fa128d8 -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-src.tar.gz) | 646edd890d6df5858b90aaf68cc6e1b4589b8db09396ae921b5c400f2188234999e6c9633906692add08c6e8b4b09f12b2099132b0a7533443fb2a01cfc2bf81 +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes.tar.gz) | c9f4f25242e319e5d90f49d26f239a930aad69677c0f3c2387c56bb13482648a26ed234be2bfe2352508f35010e3eb6d3b127c31a9f24fa1e53ac99c38520fe4 +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-src.tar.gz) | 255357db8fa160cab2187658906b674a8b0d9b9a5b5f688cc7b69dc124f5da00362c6cc18ae9b80f7ddb3da6f64c2ab2f12fb9b63a4e063c7366a5375b175cda ### Client binaries filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | c136273883e24a2a50b5093b9654f01cdfe57b97461d34885af4a68c2c4d108c07583c02b1cdf7f57f82e91306e542ce8f3bddb12fcce72b744458bc4796f8eb -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-linux-386.tar.gz) | 6ec59f1ed30569fa64ddb2d0de32b1ae04cda4ffe13f339050a7c9d7c63d425ee6f6d963dcf82c17281c4474da3eaf32c08117669052872a8c81bdce2c8a5415 -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | 7b40a4c087e2ea7f8d055f297fcd39a3f1cb6c866e7a3981a9408c3c3eb5363c648613491aad11bc7d44d5530b20832f8f96f6ceff43deede911fb74aafad35f -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | cda9955feebea5acb8f2b5b87895d24894bbbbde47041453b1f926ebdf47a258ce0496aa27d06bcbf365b5615ce68a20d659b64410c54227216726e2ee432fca -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | f65bd9241c7eb88a4886a285330f732448570aea4ededaebeabcf70d17ea185f51bf8a7218f146ee09fb1adceca7ee71fb3c3683834f2c415163add820fba96e -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | 1e377599af100a81d027d9199365fb8208d443a8e0a97affff1a79dc18796e14b78cb53d6e245c1c1e8defd0e050e37bf5f2a23c8a3ff45a6d18d03619709bf5 -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | 1cdee81478246aa7e7b80ae4efc7f070a5b058083ae278f59fad088b75a8052761b0e15ab261a6e667ddafd6a69fb424fc307072ed47941cad89a85af7aee93d -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-windows-386.tar.gz) | d8774167c87b6844c348aa15e92d5033c528d6ab9e95d08a7cb22da68bafd8e46d442cf57a5f6affad62f674c10ae6947d524b94108b5e450ca78f92656d63c0 -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | f664b47d8daa6036f8154c1dc1f881bfe683bf57c39d9b491de3848c03d051c50c6644d681baf7f9685eae45f9ce62e4c6dfea2853763cfe8256a61bdd59d894 +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-darwin-amd64.tar.gz) | 02efd389c8126456416fd2c7ea25c3cc30f612649ad91f631f068d6c0e5e539484d3763cb9a8645ad6b8077e4fcd1552a659d7516ebc4ce6828cf823b65c3016 +[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-darwin-arm64.tar.gz) | ac90dcd1699d1d7ff9c8342d481f6d0d97ccdc3ec501a56dc7c9e1898a8f77f712bf66942d304bfe581b5494f13e3efa211865de88f89749780e9e26e673dbdb +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-386.tar.gz) | cce5fb84cc7a1ee664f89d8ad3064307c51c044e9ddd2ae5a004939b69d3b3ef6f29acc5782e27d0c8f0d6d3d9c96e922f5d1b99d210ca3e754666d775df9f0c +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-amd64.tar.gz) | 2e93bbd2e60ad7cd8fe495115e96c55b1dc8facd100a827ef9c197a732679b60cceb9ea7bf92a1f5e328c3b8adfa8d3922cbc5d8370e374f3381b83f5b877b4f +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-arm.tar.gz) | 23f03b6a8fa9decce9b89a2c1bd3dae6d0b2f9e533e35a79e2c5a29326a165259677594ae83c877219a21bdb95557a284e55f4eec12954742794579c89a7d7e5 +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-arm64.tar.gz) | 3acf3101b46568b0ded6b90f13df0e918870d6812dc1a584903ddb8ba146484a204b9e442f863df47c7d4dab043fd9f7294c5510d3eb09004993d6d3b1e9e13c +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-ppc64le.tar.gz) | f749198df69577f62872d3096138a1b8969ec6b1636eb68eb56640bf33cf5f97a11df4363462749a1c0dc3ccbb8ae76c5d66864bf1c5cf7e52599caaf498e504 +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-linux-s390x.tar.gz) | 3f6c0189d59fca22cdded3a02c672ef703d17e6ab0831e173a870e14ccec436c142600e9fc35b403571b6906f2be8d18d38d33330f7caada971bbe1187b388f6 +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-windows-386.tar.gz) | 03d92371c425cf331c80807c0ac56f953be304fc6719057258a363d527d186d610e1d4b4d401b34128062983265c2e21f2d2389231aa66a6f5787eee78142cf6 +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-client-windows-amd64.tar.gz) | 489ece0c886a025ca3a25d28518637a5a824ea6544e7ef8778321036f13c8909a978ad4ceca966cec1e1cda99f25ca78bfd37460d1231c77436d216d43c872ad ### Server binaries filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | d6fcb4600be0beb9de222a8da64c35fe22798a0da82d41401d34d0f0fc7e2817512169524c281423d8f4a007cd77452d966317d5a1b67d2717a05ff346e8aa7d -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-server-linux-arm.tar.gz) | 022a76cf10801f8afbabb509572479b68fdb4e683526fa0799cdbd9bab4d3f6ecb76d1d63d0eafee93e3edf6c12892d84b9c771ef2325663b95347728fa3d6c0 -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | 0679aadd60bbf6f607e5befad74b5267eb2d4c1b55985cc25a97e0f4c5efb7acbb3ede91bfa6a5a5713dae4d7a302f6faaf678fd6b359284c33d9a6aca2a08bb -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | 9f2cfeed543b515eafb60d9765a3afff4f3d323c0a5c8a0d75e3de25985b2627817bfcbe59a9a61d969e026e2b861adb974a09eae75b58372ed736ceaaed2a82 -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | 937258704d7b9dcd91f35f2d34ee9dd38c18d9d4e867408c05281bfbbb919ad012c95880bee84d2674761aa44cc617fb2fae1124cf63b689289286d6eac1c407 +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-amd64.tar.gz) | 2e95cb31d5afcb6842c41d25b7d0c18dd7e65693b2d93c8aa44e5275f9c6201e1a67685c7a8ddefa334babb04cb559d26e39b6a18497695a07dc270568cae108 +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-arm.tar.gz) | 2927e82b98404c077196ce3968f3afd51a7576aa56d516019bd3976771c0213ba01e78da5b77478528e770da0d334e9457995fafb98820ed68b2ee34beb68856 +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-arm64.tar.gz) | e0f7aea3ea598214a9817bc04949389cb7e4e7b9503141a590ef48c0b681fe44a4243ebc6280752fa41aa1093149b3ee1bcef7664edb746097a342281825430b +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-ppc64le.tar.gz) | c011f7eb01294e9ba5d5ced719068466f88ed595dcb8d554a36a4dd5118fb6b3d6bafe8bf89aa2d42988e69793ed777ba77b8876c6ec74f898a43cfce1f61bf4 +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-server-linux-s390x.tar.gz) | 15f6683e7f16caab7eebead2b7c15799460abbf035a43de0b75f96b0be19908f58add98a777a0cca916230d60cf6bfe3fee92b9dcff50274b1e37c243c157969 ### Node binaries filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | 076165d745d47879de68f4404eaf432920884be48277eb409e84bf2c61759633bf3575f46b0995f1fc693023d76c0921ed22a01432e756d7f8d9e246a243b126 -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-node-linux-arm.tar.gz) | 1ff2e2e3e43af41118cdfb70c778e15035bbb1aca833ffd2db83c4bcd44f55693e956deb9e65017ebf3c553f2820ad5cd05f5baa33f3d63f3e00ed980ea4dfed -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | b232c7359b8c635126899beee76998078eec7a1ef6758d92bcdebe8013b0b1e4d7b33ecbf35e3f82824fe29493400845257e70ed63c1635bfa36c8b3b4969f6f -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | 51d415a068f554840f4c78d11a4fedebd7cb03c686b0ec864509b24f7a8667ebf54bb0a25debcf2b70f38be1e345e743f520695b11806539a55a3620ce21946f -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | b51c082d8af358233a088b632cf2f6c8cfe5421471c27f5dc9ba4839ae6ea75df25d84298f2042770097554c01742bb7686694b331ad9bafc93c86317b867728 -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | 91b9d26620a2dde67a0edead0039814efccbdfd54594dda3597aaced6d89140dc92612ed0727bc21d63468efeef77c845e640153b09e39d8b736062e6eee0c76 +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-amd64.tar.gz) | ed58679561197110f366b9109f7afd62c227bfc271918ccf3eea203bb2ab6428eb5db4dd6c965f202a8a636f66da199470269b863815809b99d53d2fa47af2ea +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-arm.tar.gz) | 7e6c7f1957fcdecec8fef689c5019edbc0d0c11d22dafbfef0a07121d10d8f6273644f73511bd06a9a88b04d81a940bd6645ffb5711422af64af547a45c76273 +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-arm64.tar.gz) | a3618f29967e7a1574917a67f0296e65780321eda484b99aa32bfd4dc9b35acdefce33da952ac52dfb509fbac5bf700cf177431fad2ab4adcab0544538939faa +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-ppc64le.tar.gz) | 326d3eb521b41bdf489912177f70b8cdd7cd828bb9b3d847ed3694eb27e457f24e0a88b8e51b726eee39800a3c5a40c1b30e3a8ec4a34d8041b3d8ef05d1b749 +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-linux-s390x.tar.gz) | 022d05ebaa66a0332c4fe18cdaf23d14c2c7e4d1f2af7f27baaf1eb042e6890dc3434b4ac8ba58c35d590717956f8c3458112685aff4938b94b18e263c3f4256 +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.1/kubernetes-node-windows-amd64.tar.gz) | fa691ed93f07af6bc1cf57e20a30580d6c528f88e5fea3c14f39c1820969dc5a0eb476c5b87b288593d0c086c4dd93aff6165082393283c3f46c210f9bb66d61 -## Changelog since v1.20.0-alpha.1 - -## Changes by Kind - -### Deprecation - -- Action-required: kubeadm: graduate the "kubeadm alpha certs" command to a parent command "kubeadm certs". The command "kubeadm alpha certs" is deprecated and will be removed in a future release. Please migrate. ([#94938](https://github.com/kubernetes/kubernetes/pull/94938), [@yagonobre](https://github.com/yagonobre)) [SIG Cluster Lifecycle] -- Action-required: kubeadm: remove the deprecated feature --experimental-kustomize from kubeadm commands. The feature was replaced with --experimental-patches in 1.19. To migrate see the --help description for the --experimental-patches flag. ([#94871](https://github.com/kubernetes/kubernetes/pull/94871), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: deprecate self-hosting support. The experimental command "kubeadm alpha self-hosting" is now deprecated and will be removed in a future release. ([#95125](https://github.com/kubernetes/kubernetes/pull/95125), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration ([#94884](https://github.com/kubernetes/kubernetes/pull/94884), [@arghya88](https://github.com/arghya88)) [SIG Instrumentation and Scheduling] -- Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 ([#95001](https://github.com/kubernetes/kubernetes/pull/95001), [@arghya88](https://github.com/arghya88)) [SIG Instrumentation and Scheduling] - -### API Change - -- GPU metrics provided by kubelet are now disabled by default ([#95184](https://github.com/kubernetes/kubernetes/pull/95184), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Node] -- New parameter `defaultingType` for `PodTopologySpread` plugin allows to use k8s defined or user provided default constraints ([#95048](https://github.com/kubernetes/kubernetes/pull/95048), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] -- Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. ([#93901](https://github.com/kubernetes/kubernetes/pull/93901), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing] -- Status of v1beta1 CRDs without "preserveUnknownFields:false" will show violation "spec.preserveUnknownFields: Invalid value: true: must be false" ([#93078](https://github.com/kubernetes/kubernetes/pull/93078), [@vareti](https://github.com/vareti)) [SIG API Machinery] - -### Feature - -- Added `get-users` and `delete-user` to the `kubectl config` subcommand ([#89840](https://github.com/kubernetes/kubernetes/pull/89840), [@eddiezane](https://github.com/eddiezane)) [SIG CLI] -- Added counter metric "apiserver_request_self" to count API server self-requests with labels for verb, resource, and subresource. ([#94288](https://github.com/kubernetes/kubernetes/pull/94288), [@LogicalShark](https://github.com/LogicalShark)) [SIG API Machinery, Auth, Instrumentation and Scheduling] -- Added new k8s.io/component-helpers repository providing shared helper code for (core) components. ([#92507](https://github.com/kubernetes/kubernetes/pull/92507), [@ingvagabund](https://github.com/ingvagabund)) [SIG Apps, Node, Release and Scheduling] -- Adds `create ingress` command to `kubectl` ([#78153](https://github.com/kubernetes/kubernetes/pull/78153), [@amimof](https://github.com/amimof)) [SIG CLI and Network] -- Allow configuring AWS LoadBalancer health check protocol via service annotations ([#94546](https://github.com/kubernetes/kubernetes/pull/94546), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] -- Azure: Support multiple services sharing one IP address ([#94991](https://github.com/kubernetes/kubernetes/pull/94991), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] -- Ephemeral containers now apply the same API defaults as initContainers and containers ([#94896](https://github.com/kubernetes/kubernetes/pull/94896), [@wawa0210](https://github.com/wawa0210)) [SIG Apps and CLI] -- In dual-stack bare-metal clusters, you can now pass dual-stack IPs to `kubelet --node-ip`. - eg: `kubelet --node-ip 10.1.0.5,fd01::0005`. This is not yet supported for non-bare-metal - clusters. - - In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods - will now get dual-stack PodIPs. ([#95239](https://github.com/kubernetes/kubernetes/pull/95239), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] -- Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. ([#91977](https://github.com/kubernetes/kubernetes/pull/91977), [@varunmar](https://github.com/varunmar)) [SIG Cloud Provider] -- Kubeadm: do not throw errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate. Print warnings instead. ([#94504](https://github.com/kubernetes/kubernetes/pull/94504), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags: - - apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration. - - cluster-name: use clusterName from ClusterConfiguration - - cert-dir: use certificatesDir from ClusterConfiguration ([#94879](https://github.com/kubernetes/kubernetes/pull/94879), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] -- Kubectl rollout history sts/sts-name --revision=some-revision will start showing the detailed view of the sts on that specified revision ([#86506](https://github.com/kubernetes/kubernetes/pull/86506), [@dineshba](https://github.com/dineshba)) [SIG CLI] -- Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. ([#93534](https://github.com/kubernetes/kubernetes/pull/93534), [@everpeace](https://github.com/everpeace)) [SIG Scheduling and Testing] -- Send gce node startup scripts logs to console and journal ([#95311](https://github.com/kubernetes/kubernetes/pull/95311), [@karan](https://github.com/karan)) [SIG Cloud Provider and Node] -- Support kubectl delete orphan/foreground/background options ([#93384](https://github.com/kubernetes/kubernetes/pull/93384), [@zhouya0](https://github.com/zhouya0)) [SIG CLI and Testing] - -### Bug or Regression - -- Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. ([#94916](https://github.com/kubernetes/kubernetes/pull/94916), [@smileusd](https://github.com/smileusd)) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing] -- Cloud node controller: handle empty providerID from getProviderID ([#95342](https://github.com/kubernetes/kubernetes/pull/95342), [@nicolehanjing](https://github.com/nicolehanjing)) [SIG Cloud Provider] -- Fix a bug where the endpoint slice controller was not mirroring the parent service labels to its corresponding endpoint slices ([#94443](https://github.com/kubernetes/kubernetes/pull/94443), [@aojea](https://github.com/aojea)) [SIG Apps and Network] -- Fix azure disk attach failure for disk size bigger than 4TB ([#95463](https://github.com/kubernetes/kubernetes/pull/95463), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix azure disk data loss issue on Windows when unmount disk ([#95456](https://github.com/kubernetes/kubernetes/pull/95456), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Fix detach azure disk issue when vm not exist ([#95177](https://github.com/kubernetes/kubernetes/pull/95177), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix network_programming_latency metric reporting for Endpoints/EndpointSlice deletions, where we don't have correct timestamp ([#95363](https://github.com/kubernetes/kubernetes/pull/95363), [@wojtek-t](https://github.com/wojtek-t)) [SIG Network and Scalability] -- Fix scheduler cache snapshot when a Node is deleted before its Pods ([#95130](https://github.com/kubernetes/kubernetes/pull/95130), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] -- Fix vsphere detach failure for static PVs ([#95447](https://github.com/kubernetes/kubernetes/pull/95447), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] -- Fixed a bug that prevents the use of ephemeral containers in the presence of a validating admission webhook. ([#94685](https://github.com/kubernetes/kubernetes/pull/94685), [@verb](https://github.com/verb)) [SIG Node and Testing] -- Gracefully delete nodes when their parent scale set went missing ([#95289](https://github.com/kubernetes/kubernetes/pull/95289), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] -- In dual-stack clusters, kubelet will now set up both IPv4 and IPv6 iptables rules, which may - fix some problems, eg with HostPorts. ([#94474](https://github.com/kubernetes/kubernetes/pull/94474), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] -- Kubeadm: for Docker as the container runtime, make the "kubeadm reset" command stop containers before removing them ([#94586](https://github.com/kubernetes/kubernetes/pull/94586), [@BedivereZero](https://github.com/BedivereZero)) [SIG Cluster Lifecycle] -- Kubeadm: warn but do not error out on missing "ca.key" files for root CA, front-proxy CA and etcd CA, during "kubeadm join --control-plane" if the user has provided all certificates, keys and kubeconfig files which require signing with the given CA keys. ([#94988](https://github.com/kubernetes/kubernetes/pull/94988), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Port mapping allows to map the same `containerPort` to multiple `hostPort` without naming the mapping explicitly. ([#94494](https://github.com/kubernetes/kubernetes/pull/94494), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Network and Node] -- Warn instead of fail when creating Roles and ClusterRoles with custom verbs via kubectl ([#92492](https://github.com/kubernetes/kubernetes/pull/92492), [@eddiezane](https://github.com/eddiezane)) [SIG CLI] - -### Other (Cleanup or Flake) - -- Added fine grained debugging to the intra-pod conformance test for helping easily resolve networking issues for nodes that might be unhealthy when running conformance or sonobuoy tests. ([#93837](https://github.com/kubernetes/kubernetes/pull/93837), [@jayunit100](https://github.com/jayunit100)) [SIG Network and Testing] -- AdmissionReview objects sent for the creation of Namespace API objects now populate the `namespace` attribute consistently (previously the `namespace` attribute was empty for Namespace creation via POST requests, and populated for Namespace creation via server-side-apply PATCH requests) ([#95012](https://github.com/kubernetes/kubernetes/pull/95012), [@nodo](https://github.com/nodo)) [SIG API Machinery and Testing] -- Client-go header logging (at verbosity levels >= 9) now masks `Authorization` header contents ([#95316](https://github.com/kubernetes/kubernetes/pull/95316), [@sfowl](https://github.com/sfowl)) [SIG API Machinery] -- Enhance log information of verifyRunAsNonRoot, add pod, container information ([#94911](https://github.com/kubernetes/kubernetes/pull/94911), [@wawa0210](https://github.com/wawa0210)) [SIG Node] -- Errors from staticcheck: - vendor/k8s.io/client-go/discovery/cached/memory/memcache_test.go:94:2: this value of g is never used (SA4006) ([#95098](https://github.com/kubernetes/kubernetes/pull/95098), [@phunziker](https://github.com/phunziker)) [SIG API Machinery] -- Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 ([#95419](https://github.com/kubernetes/kubernetes/pull/95419), [@jsturtevant](https://github.com/jsturtevant)) [SIG Cluster Lifecycle and Windows] -- Masks ceph RBD adminSecrets in logs when logLevel >= 4 ([#95245](https://github.com/kubernetes/kubernetes/pull/95245), [@sfowl](https://github.com/sfowl)) [SIG Storage] -- Upgrade snapshot controller to 3.0.0 ([#95412](https://github.com/kubernetes/kubernetes/pull/95412), [@saikat-royc](https://github.com/saikat-royc)) [SIG Cloud Provider] -- Remove offensive words from kubectl cluster-info command ([#95202](https://github.com/kubernetes/kubernetes/pull/95202), [@rikatz](https://github.com/rikatz)) [SIG Architecture, CLI and Testing] -- The following new metrics are available. - - network_plugin_operations_total - - network_plugin_operations_errors_total ([#93066](https://github.com/kubernetes/kubernetes/pull/93066), [@AnishShah](https://github.com/AnishShah)) [SIG Instrumentation, Network and Node] -- Vsphere: improve logging message on node cache refresh event ([#95236](https://github.com/kubernetes/kubernetes/pull/95236), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] -- `kubectl api-resources` now prints the API version (as 'API group/version', same as output of `kubectl api-versions`). The column APIGROUP is now APIVERSION ([#95253](https://github.com/kubernetes/kubernetes/pull/95253), [@sallyom](https://github.com/sallyom)) [SIG CLI] - -## Dependencies - -### Added -- github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) - -### Changed -- github.com/aws/aws-sdk-go: [v1.28.2 → v1.35.5](https://github.com/aws/aws-sdk-go/compare/v1.28.2...v1.35.5) -- github.com/jmespath/go-jmespath: [c2b33e8 → v0.4.0](https://github.com/jmespath/go-jmespath/compare/c2b33e8...v0.4.0) -- k8s.io/kube-openapi: 6aeccd4 → 8b50664 -- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.12 -- sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → b3cf1e8 - -### Removed -_Nothing has changed._ - - - -# v1.20.0-alpha.1 - - -## Downloads for v1.20.0-alpha.1 - -### Source Code - -filename | sha512 hash --------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes.tar.gz) | e7daed6502ea07816274f2371f96fe1a446d0d7917df4454b722d9eb3b5ff6163bfbbd5b92dfe7a0c1d07328b8c09c4ae966e482310d6b36de8813aaf87380b5 -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-src.tar.gz) | e91213a0919647a1215d4691a63b12d89a3e74055463a8ebd71dc1a4cabf4006b3660881067af0189960c8dab74f4a7faf86f594df69021901213ee5b56550ea - -### Client binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | 1f3add5f826fa989820d715ca38e8864b66f30b59c1abeacbb4bfb96b4e9c694eac6b3f4c1c81e0ee3451082d44828cb7515315d91ad68116959a5efbdaef1e1 -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-linux-386.tar.gz) | c62acdc8993b0a950d4b0ce0b45473bf96373d501ce61c88adf4007afb15c1d53da8d53b778a7eccac6c1624f7fdda322be9f3a8bc2d80aaad7b4237c39f5eaf -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | 1203ababfe00f9bc5be5c059324c17160a96530c1379a152db33564bbe644ccdb94b30eea15a0655bd652efb17895a46c31bbba19d4f5f473c2a0ff62f6e551f -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | 31860088596e12d739c7aed94556c2d1e217971699b950c8417a3cea1bed4e78c9ff1717b9f3943354b75b4641d4b906cd910890dbf4278287c0d224837d9a7d -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | 8d469f37fe20d6e15b5debc13cce4c22e8b7a4f6a4ac787006b96507a85ce761f63b28140d692c54b5f7deb08697f8d5ddb9bbfa8f5ac0d9241fc7de3a3fe3cd -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | 0d62ee1729cd5884946b6c73701ad3a570fa4d642190ca0fe5c1db0fb0cba9da3ac86a948788d915b9432d28ab8cc499e28aadc64530b7d549ee752a6ed93ec1 -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | 0fc0420e134ec0b8e0ab2654e1e102cebec47b48179703f1e1b79d51ee0d6da55a4e7304d8773d3cf830341ac2fe3cede1e6b0460fd88f7595534e0730422d5a -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-windows-386.tar.gz) | 3fb53b5260f4888c77c0e4ff602bbcf6bf38c364d2769850afe2b8d8e8b95f7024807c15e2b0d5603e787c46af8ac53492be9e88c530f578b8a389e3bd50c099 -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | 2f44c93463d6b5244ce0c82f147e7f32ec2233d0e29c64c3c5759e23533aebd12671bf63e986c0861e9736f9b5259bb8d138574a7c8c8efc822e35cd637416c0 - -### Server binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | ae82d14b1214e4100f0cc2c988308b3e1edd040a65267d0eddb9082409f79644e55387889e3c0904a12c710f91206e9383edf510990bee8c9ea2e297b6472551 -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-server-linux-arm.tar.gz) | 9a2a5828b7d1ddb16cc19d573e99a4af642f84129408e6203eeeb0558e7b8db77f3269593b5770b6a976fe9df4a64240ed27ad05a4bd43719e55fce1db0abf58 -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | ed700dd226c999354ce05b73927388d36d08474c15333ae689427de15de27c84feb6b23c463afd9dd81993315f31eb8265938cfc7ecf6f750247aa42b9b33fa9 -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | abb7a9d726538be3ccf5057a0c63ff9732b616e213c6ebb81363f0c49f1e168ce8068b870061ad7cba7ba1d49252f94cf00a5f68cec0f38dc8fce4e24edc5ca6 -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | 3a51888af1bfdd2d5b0101d173ee589c1f39240e4428165f5f85c610344db219625faa42f00a49a83ce943fb079be873b1a114a62003fae2f328f9bf9d1227a4 - -### Node binaries - -filename | sha512 hash --------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | d0f28e3c38ca59a7ff1bfecb48a1ce97116520355d9286afdca1200d346c10018f5bbdf890f130a388654635a2e83e908b263ed45f8a88defca52a7c1d0a7984 -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-node-linux-arm.tar.gz) | ed9d3f13028beb3be39bce980c966f82c4b39dc73beaae38cc075fea5be30b0309e555cb2af8196014f2cc9f0df823354213c314b4d6545ff6e30dd2d00ec90e -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | ad5b3268db365dcdded9a9a4bffc90c7df0f844000349accdf2b8fb5f1081e553de9b9e9fb25d5e8a4ef7252d51fa94ef94d36d2ab31d157854e164136f662c2 -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | c4de2524e513996def5eeba7b83f7b406f17eaf89d4d557833a93bd035348c81fa9375dcd5c27cfcc55d73995449fc8ee504be1b3bd7b9f108b0b2f153cb05ae -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | 9157b44e3e7bd5478af9f72014e54d1afa5cd19b984b4cd8b348b312c385016bb77f29db47f44aea08b58abf47d8a396b92a2d0e03f2fe8acdd30f4f9466cbdb -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | 8b40a43c5e6447379ad2ee8aac06e8028555e1b370a995f6001018a62411abe5fbbca6060b3d1682c5cadc07a27d49edd3204e797af46368800d55f4ca8aa1de - -## Changelog since v1.20.0-alpha.0 +## Changelog since v1.21.0-beta.0 ## Urgent Upgrade Notes ### (No, really, you MUST read this before you upgrade) - - Azure blob disk feature(`kind`: `Shared`, `Dedicated`) has been deprecated, you should use `kind`: `Managed` in `kubernetes.io/azure-disk` storage class. ([#92905](https://github.com/kubernetes/kubernetes/pull/92905), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] - - CVE-2020-8559 (Medium): Privilege escalation from compromised node to cluster. See https://github.com/kubernetes/kubernetes/issues/92914 for more details. - The API Server will no longer proxy non-101 responses for upgrade requests. This could break proxied backends (such as an extension API server) that respond to upgrade requests with a non-101 response code. ([#92941](https://github.com/kubernetes/kubernetes/pull/92941), [@tallclair](https://github.com/tallclair)) [SIG API Machinery] + - Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. ([#99471](https://github.com/kubernetes/kubernetes/pull/99471), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] + - Migrate `pkg/kubelet/(dockershim, network)` to structured logging + Exit code changed from 255 to 1 ([#98939](https://github.com/kubernetes/kubernetes/pull/98939), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Network and Node] + - Migrate `pkg/kubelet/certificate` to structured logging + Exit code changed from 255 to 1 ([#98993](https://github.com/kubernetes/kubernetes/pull/98993), [@SataQiu](https://github.com/SataQiu)) [SIG Auth and Node] + - Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99130](https://github.com/kubernetes/kubernetes/pull/99130), [@ayberk](https://github.com/ayberk)) [SIG Cloud Provider, Storage and Testing] + - Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. ([#99719](https://github.com/kubernetes/kubernetes/pull/99719), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider and Storage] + - OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. ([#98538](https://github.com/kubernetes/kubernetes/pull/98538), [@dims](https://github.com/dims)) [SIG Storage] + - Package pkg/kubelet/server migrated to structured logging + Exit code changed from 255 to 1 ([#99838](https://github.com/kubernetes/kubernetes/pull/99838), [@adisky](https://github.com/adisky)) [SIG Node] + - Pkg/kubelet/kuberuntime/kuberuntime_manager.go migrated to structured logging + Exit code changed from 255 to 1 ([#99841](https://github.com/kubernetes/kubernetes/pull/99841), [@adisky](https://github.com/adisky)) [SIG Instrumentation and Node] ## Changes by Kind ### Deprecation -- Kube-apiserver: the componentstatus API is deprecated. This API provided status of etcd, kube-scheduler, and kube-controller-manager components, but only worked when those components were local to the API server, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints. Instead of this API, etcd health is included in the kube-apiserver health check and kube-scheduler/kube-controller-manager health checks can be made directly against those components' health endpoints. ([#93570](https://github.com/kubernetes/kubernetes/pull/93570), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Cluster Lifecycle] -- Kubeadm: deprecate the "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. ([#92881](https://github.com/kubernetes/kubernetes/pull/92881), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: remove the deprecated "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. This change also removes the parent command "kubeadm alpha kubelet" as there are no more sub-commands under it for the time being. ([#94668](https://github.com/kubernetes/kubernetes/pull/94668), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: remove the deprecated --kubelet-config flag for the command "kubeadm upgrade node" ([#94869](https://github.com/kubernetes/kubernetes/pull/94869), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubelet's deprecated endpoint `metrics/resource/v1alpha1` has been removed, please adopt to `metrics/resource`. ([#94272](https://github.com/kubernetes/kubernetes/pull/94272), [@RainbowMango](https://github.com/RainbowMango)) [SIG Instrumentation and Node] -- The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. ([#94090](https://github.com/kubernetes/kubernetes/pull/94090), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing] +- Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. ([#99646](https://github.com/kubernetes/kubernetes/pull/99646), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] +- Remove deprecated --generator --replicas --service-generator --service-overrides --schedule from kubectl run + Deprecate --serviceaccount --hostport --requests --limits in kubectl run ([#99732](https://github.com/kubernetes/kubernetes/pull/99732), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing] +- `audit.k8s.io/v1beta1` and `audit.k8s.io/v1alpha1` audit policy configuration and audit events are deprecated in favor of `audit.k8s.io/v1`, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with `--audit-policy-file`, or explicitly request alpha or beta audit events with `--audit-log-version` / `--audit-webhook-version` must update to use `audit.k8s.io/v1` and accept `audit.k8s.io/v1` events prior to v1.24. ([#98858](https://github.com/kubernetes/kubernetes/pull/98858), [@carlory](https://github.com/carlory)) [SIG Auth] +- `diskformat` stroage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove `diskformat` parameter. vSphere CSI Driver does not support diskformat storageclass parameter. + + vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3. + + VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15. + + Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers. + + Support for these deprecations will be available till Kubernetes v1.24. ([#98546](https://github.com/kubernetes/kubernetes/pull/98546), [@divyenpatel](https://github.com/divyenpatel)) [SIG Cloud Provider and Storage] ### API Change -- A new `nofuzz` go build tag now disables gofuzz support. Release binaries enable this. ([#92491](https://github.com/kubernetes/kubernetes/pull/92491), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery] -- A new alpha-level field, `SupportsFsGroup`, has been introduced for CSIDrivers to allow them to specify whether they support volume ownership and permission modifications. The `CSIVolumeSupportFSGroup` feature gate must be enabled to allow this field to be used. ([#92001](https://github.com/kubernetes/kubernetes/pull/92001), [@huffmanca](https://github.com/huffmanca)) [SIG API Machinery, CLI and Storage] -- Added pod version skew strategy for seccomp profile to synchronize the deprecated annotations with the new API Server fields. Please see the corresponding section [in the KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20190717-seccomp-ga.md#version-skew-strategy) for more detailed explanations. ([#91408](https://github.com/kubernetes/kubernetes/pull/91408), [@saschagrunert](https://github.com/saschagrunert)) [SIG Apps, Auth, CLI and Node] -- Adds the ability to disable Accelerator/GPU metrics collected by Kubelet ([#91930](https://github.com/kubernetes/kubernetes/pull/91930), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Node] -- Custom Endpoints are now mirrored to EndpointSlices by a new EndpointSliceMirroring controller. ([#91637](https://github.com/kubernetes/kubernetes/pull/91637), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps, Auth, Cloud Provider, Instrumentation, Network and Testing] -- External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ ([#92632](https://github.com/kubernetes/kubernetes/pull/92632), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Node and Testing] -- Fix conversions for custom metrics. ([#94481](https://github.com/kubernetes/kubernetes/pull/94481), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] -- Generic ephemeral volumes, a new alpha feature under the `GenericEphemeralVolume` feature gate, provide a more flexible alternative to `EmptyDir` volumes: as with `EmptyDir`, volumes are created and deleted for each pod automatically by Kubernetes. But because the normal provisioning process is used (`PersistentVolumeClaim`), storage can be provided by third-party storage vendors and all of the usual volume features work. Volumes don't need to be empt; for example, restoring from snapshot is supported. ([#92784](https://github.com/kubernetes/kubernetes/pull/92784), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node, Scheduling, Storage and Testing] -- Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting `--volume-host-allow-local-loopback=false`, or from contacting specific CIDR ranges by setting `--volume-host-cidr-denylist` (for example, `--volume-host-cidr-denylist=127.0.0.1/28,feed::/16`) ([#91785](https://github.com/kubernetes/kubernetes/pull/91785), [@mattcary](https://github.com/mattcary)) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing] -- Kubernetes is now built with golang 1.15.0-rc.1. - - The deprecated, legacy behavior of treating the CommonName field on X.509 serving certificates as a host name when no Subject Alternative Names are present is now disabled by default. It can be temporarily re-enabled by adding the value x509ignoreCN=0 to the GODEBUG environment variable. ([#93264](https://github.com/kubernetes/kubernetes/pull/93264), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Storage and Testing] -- Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock ([#94603](https://github.com/kubernetes/kubernetes/pull/94603), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery, Apps, Cloud Provider and Scheduling] -- Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly ([#94182](https://github.com/kubernetes/kubernetes/pull/94182), [@mattfenwick](https://github.com/mattfenwick)) [SIG API Machinery, Apps, Auth, Network and Node] -- The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. ([#91921](https://github.com/kubernetes/kubernetes/pull/91921), [@mtaufen](https://github.com/mtaufen)) [SIG Auth] -- The kube-controller-manager managed signers can now have distinct signing certificates and keys. See the help about `--cluster-signing-[signer-name]-{cert,key}-file`. `--cluster-signing-{cert,key}-file` is still the default. ([#90822](https://github.com/kubernetes/kubernetes/pull/90822), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Apps and Auth] -- When creating a networking.k8s.io/v1 Ingress API object, `spec.tls[*].secretName` values are required to pass validation rules for Secret API object names. ([#93929](https://github.com/kubernetes/kubernetes/pull/93929), [@liggitt](https://github.com/liggitt)) [SIG Network] -- WinOverlay feature graduated to beta ([#94807](https://github.com/kubernetes/kubernetes/pull/94807), [@ksubrmnn](https://github.com/ksubrmnn)) [SIG Windows] +- 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels. + 2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. ([#98582](https://github.com/kubernetes/kubernetes/pull/98582), [@ahg-g](https://github.com/ahg-g)) [SIG API Machinery, Apps, Auth and Testing] +- Add a default metadata name labels for selecting any namespace by its name. ([#96968](https://github.com/kubernetes/kubernetes/pull/96968), [@jayunit100](https://github.com/jayunit100)) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing] +- Added `.spec.completionMode` field to Job, with accepted values `NonIndexed` (default) and `Indexed` ([#98441](https://github.com/kubernetes/kubernetes/pull/98441), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] +- Clarified NetworkPolicy policyTypes documentation ([#97216](https://github.com/kubernetes/kubernetes/pull/97216), [@joejulian](https://github.com/joejulian)) [SIG Network] +- DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. ([#96441](https://github.com/kubernetes/kubernetes/pull/96441), [@smarterclayton](https://github.com/smarterclayton)) [SIG Apps and Testing] +- EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. ([#99662](https://github.com/kubernetes/kubernetes/pull/99662), [@swetharepakula](https://github.com/swetharepakula)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network and Testing] +- EndpointSlice Controllers are now GA. The EndpointSlice Controller will not populate the `deprecatedTopology` field and will only provide topology information through the `zone` and `nodeName` fields. ([#99870](https://github.com/kubernetes/kubernetes/pull/99870), [@swetharepakula](https://github.com/swetharepakula)) [SIG API Machinery, Apps, Auth, Network and Testing] +- IngressClass resource can now reference a resource in a specific namespace + for implementation-specific configuration(previously only Cluster-level resources were allowed). + This feature can be enabled using the IngressClassNamespacedParams feature gate. ([#99275](https://github.com/kubernetes/kubernetes/pull/99275), [@hbagdi](https://github.com/hbagdi)) [SIG API Machinery, CLI and Network] +- Introduce conditions for PodDisruptionBudget ([#98127](https://github.com/kubernetes/kubernetes/pull/98127), [@mortent](https://github.com/mortent)) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation] +- Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs ([#98727](https://github.com/kubernetes/kubernetes/pull/98727), [@adtac](https://github.com/adtac)) [SIG API Machinery, Apps, Node, Scheduling and Testing] +- Kubelet Graceful Node Shutdown feature is now beta. ([#99735](https://github.com/kubernetes/kubernetes/pull/99735), [@bobbypage](https://github.com/bobbypage)) [SIG Node] +- Limit the quest value of hugepage to integer multiple of page size. ([#98515](https://github.com/kubernetes/kubernetes/pull/98515), [@lala123912](https://github.com/lala123912)) [SIG Apps] +- One new field "InternalTrafficPolicy" in Service is added. + It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. + "Cluster" routes internal traffic to a Service to all endpoints. + "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. + The default value is "Cluster". ([#96600](https://github.com/kubernetes/kubernetes/pull/96600), [@maplain](https://github.com/maplain)) [SIG API Machinery, Apps and Network] +- PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled ([#98918](https://github.com/kubernetes/kubernetes/pull/98918), [@pohly](https://github.com/pohly)) [SIG Auth and Security] +- Promote CronJobs to batch/v1 ([#99423](https://github.com/kubernetes/kubernetes/pull/99423), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] +- Remove support for building Kubernetes with bazel. ([#99561](https://github.com/kubernetes/kubernetes/pull/99561), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows] +- Setting loadBalancerClass in load balancer type of service is available with this PR. + Users who want to use a custom load balancer can specify loadBalancerClass to achieve it. ([#98277](https://github.com/kubernetes/kubernetes/pull/98277), [@XudongLiuHarold](https://github.com/XudongLiuHarold)) [SIG API Machinery, Apps, Cloud Provider and Network] +- Storage capacity tracking (= the CSIStorageCapacity feature) is beta, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated ([#99641](https://github.com/kubernetes/kubernetes/pull/99641), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Scheduling, Storage and Testing] +- Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. ([#98812](https://github.com/kubernetes/kubernetes/pull/98812), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and CLI] +- The apiserver now resets managedFields that got corrupted by a mutating admission controller. ([#98074](https://github.com/kubernetes/kubernetes/pull/98074), [@kwiesmueller](https://github.com/kwiesmueller)) [SIG API Machinery and Testing] +- `controller.kubernetes.io/pod-deletion-cost` annotation can be set to offer a hint on the cost of deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. ([#99163](https://github.com/kubernetes/kubernetes/pull/99163), [@ahg-g](https://github.com/ahg-g)) [SIG Apps] ### Feature -- ACTION REQUIRED : In CoreDNS v1.7.0, [metrics names have been changed](https://github.com/coredns/coredns/blob/master/notes/coredns-1.7.0.md#metric-changes) which will be backward incompatible with existing reporting formulas that use the old metrics' names. Adjust your formulas to the new names before upgrading. - - Kubeadm now includes CoreDNS version v1.7.0. Some of the major changes include: - - Fixed a bug that could cause CoreDNS to stop updating service records. - - Fixed a bug in the forward plugin where only the first upstream server is always selected no matter which policy is set. - - Remove already deprecated options `resyncperiod` and `upstream` in the Kubernetes plugin. - - Includes Prometheus metrics name changes (to bring them in line with standard Prometheus metrics naming convention). They will be backward incompatible with existing reporting formulas that use the old metrics' names. - - The federation plugin (allows for v1 Kubernetes federation) has been removed. - More details are available in https://coredns.io/2020/06/15/coredns-1.7.0-release/ ([#92651](https://github.com/kubernetes/kubernetes/pull/92651), [@rajansandeep](https://github.com/rajansandeep)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation] -- Add metrics for azure service operations (route and loadbalancer). ([#94124](https://github.com/kubernetes/kubernetes/pull/94124), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider and Instrumentation] -- Add network rule support in Azure account creation ([#94239](https://github.com/kubernetes/kubernetes/pull/94239), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Add tags support for Azure File Driver ([#92825](https://github.com/kubernetes/kubernetes/pull/92825), [@ZeroMagic](https://github.com/ZeroMagic)) [SIG Cloud Provider and Storage] -- Added kube-apiserver metrics: apiserver_current_inflight_request_measures and, when API Priority and Fairness is enable, windowed_request_stats. ([#91177](https://github.com/kubernetes/kubernetes/pull/91177), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery, Instrumentation and Testing] -- Audit events for API requests to deprecated API versions now include a `"k8s.io/deprecated": "true"` audit annotation. If a target removal release is identified, the audit event includes a `"k8s.io/removal-release": "."` audit annotation as well. ([#92842](https://github.com/kubernetes/kubernetes/pull/92842), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Instrumentation] -- Cloud node-controller use InstancesV2 ([#91319](https://github.com/kubernetes/kubernetes/pull/91319), [@gongguan](https://github.com/gongguan)) [SIG Apps, Cloud Provider, Scalability and Storage] -- Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM ([#93275](https://github.com/kubernetes/kubernetes/pull/93275), [@xlgao-zju](https://github.com/xlgao-zju)) [SIG Cluster Lifecycle] -- Kubeadm: add the "--cluster-name" flag to the "kubeadm alpha kubeconfig user" to allow configuring the cluster name in the generated kubeconfig file ([#93992](https://github.com/kubernetes/kubernetes/pull/93992), [@prabhu43](https://github.com/prabhu43)) [SIG Cluster Lifecycle] -- Kubeadm: add the "--kubeconfig" flag to the "kubeadm init phase upload-certs" command to allow users to pass a custom location for a kubeconfig file. ([#94765](https://github.com/kubernetes/kubernetes/pull/94765), [@zhanw15](https://github.com/zhanw15)) [SIG Cluster Lifecycle] -- Kubeadm: deprecate the "--csr-only" and "--csr-dir" flags of the "kubeadm init phase certs" subcommands. Please use "kubeadm alpha certs generate-csr" instead. This new command allows you to generate new private keys and certificate signing requests for all the control-plane components, so that the certificates can be signed by an external CA. ([#92183](https://github.com/kubernetes/kubernetes/pull/92183), [@wallrj](https://github.com/wallrj)) [SIG Cluster Lifecycle] -- Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default ([#94479](https://github.com/kubernetes/kubernetes/pull/94479), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] -- Kubemark now supports both real and hollow nodes in a single cluster. ([#93201](https://github.com/kubernetes/kubernetes/pull/93201), [@ellistarn](https://github.com/ellistarn)) [SIG Scalability] -- Kubernetes is now built using go1.15.2 - - build: Update to k/repo-infra@v0.1.1 (supports go1.15.2) - - build: Use go-runner:buster-v2.0.1 (built using go1.15.1) - - bazel: Replace --features with Starlark build settings flag - - hack/lib/util.sh: some bash cleanups - - - switched one spot to use kube::logging - - make kube::util::find-binary return an error when it doesn't find - anything so that hack scripts fail fast instead of with '' binary not - found errors. - - this required deleting some genfeddoc stuff. the binary no longer - exists in k/k repo since we removed federation/, and I don't see it - in https://github.com/kubernetes-sigs/kubefed/ either. I'm assuming - that it's gone for good now. - - - bazel: output go_binary rule directly from go_binary_conditional_pure - - From: @mikedanese: - Instead of aliasing. Aliases are annoying in a number of ways. This is - specifically bugging me now because they make the action graph harder to - analyze programmatically. By using aliases here, we would need to handle - potentially aliased go_binary targets and dereference to the effective - target. - - The comment references an issue with `pure = select(...)` which appears - to be resolved considering this now builds. - - - make kube::util::find-binary not dependent on bazel-out/ structure - - Implement an aspect that outputs go_build_mode metadata for go binaries, - and use that during binary selection. ([#94449](https://github.com/kubernetes/kubernetes/pull/94449), [@justaugustus](https://github.com/justaugustus)) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing] -- Only update Azure data disks when attach/detach ([#94265](https://github.com/kubernetes/kubernetes/pull/94265), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Promote SupportNodePidsLimit to GA to provide node to pod pid isolation - Promote SupportPodPidsLimit to GA to provide ability to limit pids per pod ([#94140](https://github.com/kubernetes/kubernetes/pull/94140), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node and Testing] -- Rename pod_preemption_metrics to preemption_metrics. ([#93256](https://github.com/kubernetes/kubernetes/pull/93256), [@ahg-g](https://github.com/ahg-g)) [SIG Instrumentation and Scheduling] -- Server-side apply behavior has been regularized in the case where a field is removed from the applied configuration. Removed fields which have no other owners are deleted from the live object, or reset to their default value if they have one. Safe ownership transfers, such as the transfer of a `replicas` field from a user to an HPA without resetting to the default value are documented in [Transferring Ownership](/docs/reference/using-api/server-side-apply/#transferring-ownership) ([#92661](https://github.com/kubernetes/kubernetes/pull/92661), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Testing] -- Set CSIMigrationvSphere feature gates to beta. - Users should enable CSIMigration + CSIMigrationvSphere features and install the vSphere CSI Driver (https://github.com/kubernetes-sigs/vsphere-csi-driver) to move workload from the in-tree vSphere plugin "kubernetes.io/vsphere-volume" to vSphere CSI Driver. - - Requires: vSphere vCenter/ESXi Version: 7.0u1, HW Version: VM version 15 ([#92816](https://github.com/kubernetes/kubernetes/pull/92816), [@divyenpatel](https://github.com/divyenpatel)) [SIG Cloud Provider and Storage] -- Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] ([#94114](https://github.com/kubernetes/kubernetes/pull/94114), [@MarcPow](https://github.com/MarcPow)) [SIG Cloud Provider] -- Support a smooth upgrade from client-side apply to server-side apply without conflicts, as well as support the corresponding downgrade. ([#90187](https://github.com/kubernetes/kubernetes/pull/90187), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG API Machinery and Testing] -- Trace output in apiserver logs is more organized and comprehensive. Traces are nested, and for all non-long running request endpoints, the entire filter chain is instrumented (e.g. authentication check is included). ([#88936](https://github.com/kubernetes/kubernetes/pull/88936), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Scheduling] -- `kubectl alpha debug` now supports debugging nodes by creating a debugging container running in the node's host namespaces. ([#92310](https://github.com/kubernetes/kubernetes/pull/92310), [@verb](https://github.com/verb)) [SIG CLI] +- A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. ([#98892](https://github.com/kubernetes/kubernetes/pull/98892), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation] +- Add --use-protocol-buffers flag to kubectl top pods and nodes ([#96655](https://github.com/kubernetes/kubernetes/pull/96655), [@serathius](https://github.com/serathius)) [SIG CLI] +- Add support to generate client-side binaries for new darwin/arm64 platform ([#97743](https://github.com/kubernetes/kubernetes/pull/97743), [@dims](https://github.com/dims)) [SIG Release and Testing] +- Added `ephemeral_volume_controller_create[_failures]_total` counters to kube-controller-manager metrics ([#99115](https://github.com/kubernetes/kubernetes/pull/99115), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage] +- Adds alpha feature `VolumeCapacityPriority` which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. ([#96347](https://github.com/kubernetes/kubernetes/pull/96347), [@cofyc](https://github.com/cofyc)) [SIG Apps, Network, Scheduling, Storage and Testing] +- Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, and a gauge for the missed schedules of a cronjob ([#99341](https://github.com/kubernetes/kubernetes/pull/99341), [@alaypatel07](https://github.com/alaypatel07)) [SIG Apps and Instrumentation] +- Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. + - To enable: export KUBECTL_COMMAND_HEADERS=1; kubectl ... ([#98952](https://github.com/kubernetes/kubernetes/pull/98952), [@seans3](https://github.com/seans3)) [SIG API Machinery and CLI] +- Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. ([#99738](https://github.com/kubernetes/kubernetes/pull/99738), [@YoyinZyc](https://github.com/YoyinZyc)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] +- Disruption controller only sends one event per PodDisruptionBudget if scale can't be computed ([#98128](https://github.com/kubernetes/kubernetes/pull/98128), [@mortent](https://github.com/mortent)) [SIG Apps] +- EndpointSliceNodeName will always be enabled, so NodeName will always be available in the v1beta1 API. ([#99746](https://github.com/kubernetes/kubernetes/pull/99746), [@swetharepakula](https://github.com/swetharepakula)) [SIG Apps and Network] +- Graduate CRIContainerLogRotation feature gate to GA. ([#99651](https://github.com/kubernetes/kubernetes/pull/99651), [@umohnani8](https://github.com/umohnani8)) [SIG Node and Testing] +- Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration ([#99653](https://github.com/kubernetes/kubernetes/pull/99653), [@aojea](https://github.com/aojea)) [SIG Instrumentation and Network] +- Kube-scheduler now logs plugin scoring summaries at --v=4 ([#99411](https://github.com/kubernetes/kubernetes/pull/99411), [@damemi](https://github.com/damemi)) [SIG Scheduling] +- Kubeadm: a warning to user as ipv6 site-local is deprecated ([#99574](https://github.com/kubernetes/kubernetes/pull/99574), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle and Network] +- Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. ([#98269](https://github.com/kubernetes/kubernetes/pull/98269), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. ([#99476](https://github.com/kubernetes/kubernetes/pull/99476), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Kubeadm: promote IPv6DualStack feature gate to Beta ([#99294](https://github.com/kubernetes/kubernetes/pull/99294), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] +- Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. ([#98250](https://github.com/kubernetes/kubernetes/pull/98250), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] +- Kubernetes is now built with Golang 1.16 ([#98572](https://github.com/kubernetes/kubernetes/pull/98572), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing] +- Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. ([#99361](https://github.com/kubernetes/kubernetes/pull/99361), [@Novex](https://github.com/Novex)) [SIG Storage] +- Remove cAdvisor json metrics api collected by Kubelet ([#99236](https://github.com/kubernetes/kubernetes/pull/99236), [@pacoxu](https://github.com/pacoxu)) [SIG Node] +- Sysctls is now GA and locked to default ([#99158](https://github.com/kubernetes/kubernetes/pull/99158), [@wgahnagl](https://github.com/wgahnagl)) [SIG Node] +- The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. ([#99213](https://github.com/kubernetes/kubernetes/pull/99213), [@AliceZhang2016](https://github.com/AliceZhang2016)) [SIG Scheduling] +- The endpointslice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. + The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. ([#98116](https://github.com/kubernetes/kubernetes/pull/98116), [@aojea](https://github.com/aojea)) [SIG Apps, Network and Testing] +- Update the latest validated version of Docker to 20.10 ([#98977](https://github.com/kubernetes/kubernetes/pull/98977), [@neolit123](https://github.com/neolit123)) [SIG CLI, Cluster Lifecycle and Node] +- Upgrade node local dns to 1.17.0 for better IPv6 support ([#99749](https://github.com/kubernetes/kubernetes/pull/99749), [@pacoxu](https://github.com/pacoxu)) [SIG Cloud Provider and Network] +- Users might specify the `kubectl.kubernetes.io/default-exec-container` annotation in a Pod to preselect container for kubectl commands. ([#99581](https://github.com/kubernetes/kubernetes/pull/99581), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG CLI] +- When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. ([#99212](https://github.com/kubernetes/kubernetes/pull/99212), [@damemi](https://github.com/damemi)) [SIG Apps and Testing] +- When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables + for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU + and memory usage of the kube-apiserver in large clusters. ([#99393](https://github.com/kubernetes/kubernetes/pull/99393), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery, Node and Testing] +- WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. ([#99794](https://github.com/kubernetes/kubernetes/pull/99794), [@robscott](https://github.com/robscott)) [SIG Network] + +### Bug or Regression + +- Creating a PVC with DataSource should fail for non-CSI plugins. ([#97086](https://github.com/kubernetes/kubernetes/pull/97086), [@xing-yang](https://github.com/xing-yang)) [SIG Apps and Storage] +- EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99345](https://github.com/kubernetes/kubernetes/pull/99345), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. ([#99756](https://github.com/kubernetes/kubernetes/pull/99756), [@robscott](https://github.com/robscott)) [SIG Apps and Network] +- Fix --ignore-errors does not take effect if multiple logs are printed and unfollowed ([#97686](https://github.com/kubernetes/kubernetes/pull/97686), [@wzshiming](https://github.com/wzshiming)) [SIG CLI] +- Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid ([#99514](https://github.com/kubernetes/kubernetes/pull/99514), [@mikkeloscar](https://github.com/mikkeloscar)) [SIG Apps and Autoscaling] +- Fix cgroup handling for systemd with cgroup v2 ([#98365](https://github.com/kubernetes/kubernetes/pull/98365), [@odinuge](https://github.com/odinuge)) [SIG Node] +- Fix smb mount PermissionDenied issue on Windows ([#99550](https://github.com/kubernetes/kubernetes/pull/99550), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider, Storage and Windows] +- Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) ([#99613](https://github.com/kubernetes/kubernetes/pull/99613), [@xh4n3](https://github.com/xh4n3)) [SIG Network] +- Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. ([#99315](https://github.com/kubernetes/kubernetes/pull/99315), [@iwankgb](https://github.com/iwankgb)) [SIG Node] +- Fixes add-on manager leader election ([#98968](https://github.com/kubernetes/kubernetes/pull/98968), [@liggitt](https://github.com/liggitt)) [SIG Cloud Provider] +- Improved update time of pod statuses following new probe results. ([#98376](https://github.com/kubernetes/kubernetes/pull/98376), [@matthyx](https://github.com/matthyx)) [SIG Node and Testing] +- Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume ([#99446](https://github.com/kubernetes/kubernetes/pull/99446), [@pohly](https://github.com/pohly)) [SIG Apps, Node and Storage] +- Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false ([#98984](https://github.com/kubernetes/kubernetes/pull/98984), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] +- On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report `ipFamilyPolicy RequireDualStack` and will have entries in `ipFamilies[]` for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. ([#99555](https://github.com/kubernetes/kubernetes/pull/99555), [@thockin](https://github.com/thockin)) [SIG Apps and Network] +- Resolves spurious `Failed to list *v1.Secret` or `Failed to list *v1.ConfigMap` messages in kubelet logs. ([#99538](https://github.com/kubernetes/kubernetes/pull/99538), [@liggitt](https://github.com/liggitt)) [SIG Auth and Node] +- Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. ([#99585](https://github.com/kubernetes/kubernetes/pull/99585), [@Iceber](https://github.com/Iceber)) [SIG Node] +- Stdin is now only passed to client-go exec credential plugins when it is detected to be an interactive terminal. Previously, it was passed to client-go exec plugins when **stdout*- was detected to be an interactive terminal. ([#99654](https://github.com/kubernetes/kubernetes/pull/99654), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery and Auth] +- The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 ([#99795](https://github.com/kubernetes/kubernetes/pull/99795), [@robscott](https://github.com/robscott)) [SIG Network] +- Updates the commands + - kubectl kustomize {arg} + - kubectl apply -k {arg} + to use same code as kustomize CLI v4.0.5 + - [v4.0.5]: https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.0.5 ([#98946](https://github.com/kubernetes/kubernetes/pull/98946), [@monopole](https://github.com/monopole)) [SIG API Machinery, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage] +- When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the + "primary IP family" of the cluster by picking a primary pod IP of the same family + as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs + in the order the administrator wanted (since some CNI plugins don't allow + configuring this). ([#97979](https://github.com/kubernetes/kubernetes/pull/97979), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] +- When using Containerd on Windows, the "C:\Windows\System32\drivers\etc\hosts" file will now be managed by kubelet. ([#83730](https://github.com/kubernetes/kubernetes/pull/83730), [@claudiubelu](https://github.com/claudiubelu)) [SIG Node and Windows] +- `VolumeBindingArgs` now allow `BindTimeoutSeconds` to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. ([#99835](https://github.com/kubernetes/kubernetes/pull/99835), [@chendave](https://github.com/chendave)) [SIG Scheduling and Storage] +- `kubectl exec` and `kubectl attach` now honor the `--quiet` flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. ([#99004](https://github.com/kubernetes/kubernetes/pull/99004), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] + +### Other (Cleanup or Flake) + +- Apiserver_request_duration_seconds is promoted to stable status. ([#99925](https://github.com/kubernetes/kubernetes/pull/99925), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] +- Apiserver_request_total is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. ([#99788](https://github.com/kubernetes/kubernetes/pull/99788), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Instrumentation and Testing] +- EndpointSlice generation is now incremented when labels change. ([#99750](https://github.com/kubernetes/kubernetes/pull/99750), [@robscott](https://github.com/robscott)) [SIG Network] +- Featuregate AllowInsecureBackendProxy is promoted to GA ([#99658](https://github.com/kubernetes/kubernetes/pull/99658), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] +- Migrate `pkg/kubelet/(eviction)` to structured logging ([#99032](https://github.com/kubernetes/kubernetes/pull/99032), [@yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Node] +- Migrate deployment controller log messages to structured logging ([#97507](https://github.com/kubernetes/kubernetes/pull/97507), [@aldudko](https://github.com/aldudko)) [SIG Apps] +- Migrate pkg/kubelet/cloudresource to structured logging ([#98999](https://github.com/kubernetes/kubernetes/pull/98999), [@sladyn98](https://github.com/sladyn98)) [SIG Node] +- Migrate pkg/kubelet/cri/remote logs to structured logging ([#98589](https://github.com/kubernetes/kubernetes/pull/98589), [@chenyw1990](https://github.com/chenyw1990)) [SIG Node] +- Migrate pkg/kubelet/kuberuntime/kuberuntime_container.go logs to structured logging ([#96973](https://github.com/kubernetes/kubernetes/pull/96973), [@chenyw1990](https://github.com/chenyw1990)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/status to structured logging ([#99836](https://github.com/kubernetes/kubernetes/pull/99836), [@navidshaikh](https://github.com/navidshaikh)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/token to structured logging ([#99264](https://github.com/kubernetes/kubernetes/pull/99264), [@palnabarun](https://github.com/palnabarun)) [SIG Auth, Instrumentation and Node] +- Migrate pkg/kubelet/util to structured logging ([#99823](https://github.com/kubernetes/kubernetes/pull/99823), [@navidshaikh](https://github.com/navidshaikh)) [SIG Instrumentation and Node] +- Migrate proxy/userspace/proxier.go logs to structured logging ([#97837](https://github.com/kubernetes/kubernetes/pull/97837), [@JornShen](https://github.com/JornShen)) [SIG Network] +- Migrate some kubelet/metrics log messages to structured logging ([#98627](https://github.com/kubernetes/kubernetes/pull/98627), [@jialaijun](https://github.com/jialaijun)) [SIG Instrumentation and Node] +- Process start time on Windows now uses current process information ([#97491](https://github.com/kubernetes/kubernetes/pull/97491), [@jsturtevant](https://github.com/jsturtevant)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows] + +### Uncategorized + +- Migrate pkg/kubelet/stats to structured logging ([#99607](https://github.com/kubernetes/kubernetes/pull/99607), [@krzysiekg](https://github.com/krzysiekg)) [SIG Node] +- The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. ([#99610](https://github.com/kubernetes/kubernetes/pull/99610), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node] + +## Dependencies + +### Added +- github.com/go-errors/errors: [v1.0.1](https://github.com/go-errors/errors/tree/v1.0.1) +- github.com/gobuffalo/here: [v0.6.0](https://github.com/gobuffalo/here/tree/v0.6.0) +- github.com/google/shlex: [e7afc7f](https://github.com/google/shlex/tree/e7afc7f) +- github.com/markbates/pkger: [v0.17.1](https://github.com/markbates/pkger/tree/v0.17.1) +- github.com/monochromegane/go-gitignore: [205db1a](https://github.com/monochromegane/go-gitignore/tree/205db1a) +- github.com/niemeyer/pretty: [a10e7ca](https://github.com/niemeyer/pretty/tree/a10e7ca) +- github.com/xlab/treeprint: [a009c39](https://github.com/xlab/treeprint/tree/a009c39) +- go.starlark.net: 8dd3e2e +- golang.org/x/term: 6a3ed07 +- sigs.k8s.io/kustomize/api: v0.8.5 +- sigs.k8s.io/kustomize/cmd/config: v0.9.7 +- sigs.k8s.io/kustomize/kustomize/v4: v4.0.5 +- sigs.k8s.io/kustomize/kyaml: v0.10.15 + +### Changed +- dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891 +- github.com/creack/pty: [v1.1.7 → v1.1.9](https://github.com/creack/pty/compare/v1.1.7...v1.1.9) +- github.com/go-openapi/spec: [v0.19.3 → v0.19.5](https://github.com/go-openapi/spec/compare/v0.19.3...v0.19.5) +- github.com/go-openapi/strfmt: [v0.19.3 → v0.19.5](https://github.com/go-openapi/strfmt/compare/v0.19.3...v0.19.5) +- github.com/go-openapi/validate: [v0.19.5 → v0.19.8](https://github.com/go-openapi/validate/compare/v0.19.5...v0.19.8) +- github.com/google/cadvisor: [v0.38.7 → v0.38.8](https://github.com/google/cadvisor/compare/v0.38.7...v0.38.8) +- github.com/kr/text: [v0.1.0 → v0.2.0](https://github.com/kr/text/compare/v0.1.0...v0.2.0) +- github.com/mattn/go-runewidth: [v0.0.2 → v0.0.7](https://github.com/mattn/go-runewidth/compare/v0.0.2...v0.0.7) +- github.com/olekukonko/tablewriter: [a0225b3 → v0.0.4](https://github.com/olekukonko/tablewriter/compare/a0225b3...v0.0.4) +- github.com/sergi/go-diff: [v1.0.0 → v1.1.0](https://github.com/sergi/go-diff/compare/v1.0.0...v1.1.0) +- golang.org/x/crypto: 7f63de1 → 5ea612d +- golang.org/x/exp: 6cc2880 → 85be41e +- golang.org/x/mobile: d2bd2a2 → e6ae53a +- golang.org/x/mod: v0.3.0 → ce943fd +- golang.org/x/net: 69a7880 → 3d97a24 +- golang.org/x/sys: 5cba982 → a50acf3 +- golang.org/x/time: 3af7569 → f8bda1e +- golang.org/x/tools: 113979e → v0.1.0 +- gopkg.in/check.v1: 41f04d3 → 8fa4692 +- gopkg.in/yaml.v2: v2.2.8 → v2.4.0 +- k8s.io/kube-openapi: d219536 → 591a79e +- k8s.io/system-validators: v1.3.0 → v1.4.0 + +### Removed +- github.com/codegangsta/negroni: [v1.0.0](https://github.com/codegangsta/negroni/tree/v1.0.0) +- github.com/golangplus/bytes: [45c989f](https://github.com/golangplus/bytes/tree/45c989f) +- github.com/golangplus/fmt: [2a5d6d7](https://github.com/golangplus/fmt/tree/2a5d6d7) +- github.com/gorilla/context: [v1.1.1](https://github.com/gorilla/context/tree/v1.1.1) +- github.com/kr/pty: [v1.1.5](https://github.com/kr/pty/tree/v1.1.5) +- sigs.k8s.io/kustomize: v2.0.3+incompatible + + + +# v1.21.0-beta.0 + + +## Downloads for v1.21.0-beta.0 + +### Source Code + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes.tar.gz) | 69b73a03b70b0ed006e9fef3f5b9bc68f0eb8dc40db6cc04777c03a2cb83a008c783012ca186b1c48357fb192403dbcf6960f120924785e2076e215b9012d546 +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-src.tar.gz) | 9620fb6d37634271bdd423c09f33f3bd29e74298aa82c47dffc8cb6bd2ff44fa8987a53c53bc529db4ca96ec41503aa81cc8d0c3ac106f3b06c4720de933a8e6 + +### Client binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-darwin-amd64.tar.gz) | 2a6f3fcd6b571f5ccde56b91e6e179a01899244be496dae16a2a16e0405c9437b75c6dc853b56f9a4876a7c0a60ec624ccd28400bf8fb960258263172f6860ba +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-386.tar.gz) | 78fe9ad9f9a9bc043293327223f0038a2c087ca65e87187a6dcae7a24aef9565fe498d295a4639b0b90524469a04930022fcecd815d0afc742eb87ddd8eb7ef5 +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-amd64.tar.gz) | c025f5e5bd132355e7dd1296cf2ec752264e7f754c4d95fc34b076bd75bef2f571d30872bcb3d138ce95c592111353d275a80eb31f82c07000874b4c56282dbd +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-arm.tar.gz) | 9975cd2f08fbc202575fb15ba6fc51dab23155ca4d294ebb48516a81efa51f58bab3a87d41c865103756189b554c020371d729ad42880ba788f25047ffc46910 +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-arm64.tar.gz) | 56a6836e24471e42e9d9a8488453f2d55598d70c8aca0a307d5116139c930c25c469fd0d1ab5060fbe88dad75a9b5209a08dc11d644af5f3ebebfbcb6c16266c +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-ppc64le.tar.gz) | b6a6cc9baad0ad85ed079ee80e6d6acc905095cfb440998bbc0f553b94fa80077bd58b8692754de477517663d51161705e6e89a1b6d04aa74819800db3517722 +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-linux-s390x.tar.gz) | 7b743481b340f510bf9ae28ea8ea91150aa1e8c37fe104b66d7b3aff62f5e6db3c590d2c13d14dbb5c928de31c7613372def2496075853611d10d6b5fa5b60bd +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-windows-386.tar.gz) | df06c7a524ce84c1f8d7836aa960c550c88dbca0ec4854df4dd0a85b3c84b8ecbc41b54e8c4669ce28ac670659ff0fad795deb1bc539f3c3b3aa885381265f5a +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-client-windows-amd64.tar.gz) | 4568497b684564f2a94fbea6cbfd778b891231470d9a6956c3b7a3268643d13b855c0fc5ebea5f769300cc0c7719c2c331c387f468816f182f63e515adeaa7a0 + +### Server binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-amd64.tar.gz) | 42883cca2d312153baf693fc6024a295359a421e74fd70eefc927413be4e0353debe634e7cca6b9a8f7d8a0cee3717e03ba5d29a306e93139b1c2f3027535a6d +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-arm.tar.gz) | e0042215e84c769ba4fc4d159ccf67b2c4a26206bfffb0ec5152723dc813ff9c1426aa0e9b963d7bfa2efb266ca43561b596b459152882ebb42102ccf60bd8eb +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-arm64.tar.gz) | bfad29d43e14152cb9bc7c4df6aa77929c6eca64a294bb832215bdba9fa0ee2195a2b709c0267dc7426bb371b547ee80bb8461a8c678c9bffa0819aa7db96289 +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-ppc64le.tar.gz) | ca67674c01c6cebdc8160c85b449eab1a23bb0557418665246e0208543fa2eaaf97679685c7b49bee3a4300904c0399c3d762ae34dc3e279fd69ce792c4b07ff +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-server-linux-s390x.tar.gz) | 285352b628ec754b01b8ad4ef1427223a142d58ebcb46f6861df14d68643133b32330460b213b1ba5bc5362ff2b6dacd8e0c2d20cce6e760fa1954af8a60df8b + +### Node binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-amd64.tar.gz) | d92d9b30e7e44134a0cd9db4c01924d365991ea16b3131200b02a82cff89c8701f618cd90e7f1c65427bd4bb5f78b10d540b2262de2c143b401fa44e5b25627b +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-arm.tar.gz) | 551092f23c27fdea4bb2d0547f6075892534892a96fc2be7786f82b58c93bffdb5e1c20f8f11beb8bed46c24f36d4c18ec5ac9755435489efa28e6ae775739bd +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-arm64.tar.gz) | 26ae7f4163e527349b8818ee38b9ee062314ab417f307afa49c146df8f5a2bd689509b128bd4a1efd3896fd89571149a9955ada91f8ca0c2f599cd863d613c86 +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-ppc64le.tar.gz) | 821fa953f6cebc69d2d481e489f3e90899813d20e2eefbabbcadd019d004108e7540f741fabe60e8e7c6adbb1053ac97898bbdddec3ca19f34a71aa3312e0d4e +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-linux-s390x.tar.gz) | 22197d4f66205d5aa9de83dfddcc4f2bb3195fd7067cdb5c21e61dbeae217bc112fb7ecff8a539579b60ad92298c2b4c87b9b7c7e6ec1ee1ffa0c6e4bc4412c1 +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-beta.0/kubernetes-node-windows-amd64.tar.gz) | 7e22e0d9603562a04dee16a513579f06b1ff6354d97d669bd68f8777ec7f89f6ef027fb23ab0445d7bba0bb689352f0cc748ce90e3f597c6ebe495464a96b860 + +## Changelog since v1.21.0-alpha.3 + +## Urgent Upgrade Notes + +### (No, really, you MUST read this before you upgrade) + + - The metric `storage_operation_errors_total` is not removed, but is marked deprecated, and the metric `storage_operation_status_count` is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using `status=fail-unknown` in the case of `storage_operations_errors_total`). ([#99045](https://github.com/kubernetes/kubernetes/pull/99045), [@mattcary](https://github.com/mattcary)) [SIG Instrumentation and Storage] + +## Changes by Kind + +### Deprecation + +- The `batch/v2alpha1` CronJob type definitions and clients are deprecated and removed. ([#96987](https://github.com/kubernetes/kubernetes/pull/96987), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, Apps, CLI and Testing] + +### API Change + +- Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in their kubelet configuration file. enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. ([#98458](https://github.com/kubernetes/kubernetes/pull/98458), [@SaranBalaji90](https://github.com/SaranBalaji90)) [SIG Node] +- The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default. + - This changes the tokens provided to containers at `/var/run/secrets/kubernetes.io/serviceaccount/token` to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted. + - Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. `k8s.io/client-go` version v11.0.0+ and v0.15.0+ reload tokens automatically. + - By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric `serviceaccount_stale_tokens_total` can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting `kube-apiserver` with `--service-account-extend-token-expiration=false`. ([#95667](https://github.com/kubernetes/kubernetes/pull/95667), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle and Testing] + +### Feature + +- A new histogram metric to track the time it took to delete a job by the ttl-after-finished controller ([#98676](https://github.com/kubernetes/kubernetes/pull/98676), [@ahg-g](https://github.com/ahg-g)) [SIG Apps and Instrumentation] +- AWS cloudprovider supports auto-discovering subnets without any kubernetes.io/cluster/ tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. ([#97431](https://github.com/kubernetes/kubernetes/pull/97431), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] +- Add --permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoid waiting for the kernel to release socket in TIME_WAIT state, and hence, considably reducing kube-apiserver restart times under certain conditions. ([#93861](https://github.com/kubernetes/kubernetes/pull/93861), [@sttts](https://github.com/sttts)) [SIG API Machinery] +- Add `csi_operations_seconds` metric on kubelet that exposes CSI operations duration and status for node CSI operations. ([#98979](https://github.com/kubernetes/kubernetes/pull/98979), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Instrumentation and Storage] +- Add `migrated` field into `storage_operation_duration_seconds` metric ([#99050](https://github.com/kubernetes/kubernetes/pull/99050), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Instrumentation and Storage] +- Add bash-completion for comma separated list on `kubectl get` ([#98301](https://github.com/kubernetes/kubernetes/pull/98301), [@phil9909](https://github.com/phil9909)) [SIG CLI] +- Added support for installing arm64 node artifacts. ([#99242](https://github.com/kubernetes/kubernetes/pull/99242), [@liu-cong](https://github.com/liu-cong)) [SIG Cloud Provider] +- Feature gate RootCAConfigMap is graduated to GA in 1.21 and will be removed in 1.22. ([#98033](https://github.com/kubernetes/kubernetes/pull/98033), [@zshihang](https://github.com/zshihang)) [SIG API Machinery and Auth] +- Kubeadm: during "init" and "join" perform preflight validation on the host / node name and throw warnings if a name is not compliant ([#99194](https://github.com/kubernetes/kubernetes/pull/99194), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] +- Kubectl: `kubectl get` will omit managed fields by default now. Users could set `--show-managed-fields` to true to show managedFields when the output format is either `json` or `yaml`. ([#96878](https://github.com/kubernetes/kubernetes/pull/96878), [@knight42](https://github.com/knight42)) [SIG CLI and Testing] +- Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=bad_metric1,bad_metric2') ([#99217](https://github.com/kubernetes/kubernetes/pull/99217), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] +- TTLAfterFinished is now beta and enabled by default ([#98678](https://github.com/kubernetes/kubernetes/pull/98678), [@ahg-g](https://github.com/ahg-g)) [SIG Apps and Auth] +- The `RunAsGroup` feature has been promoted to GA in this release. ([#94641](https://github.com/kubernetes/kubernetes/pull/94641), [@krmayankk](https://github.com/krmayankk)) [SIG Auth and Node] +- Turn CronJobControllerV2 on by default. ([#98878](https://github.com/kubernetes/kubernetes/pull/98878), [@soltysh](https://github.com/soltysh)) [SIG Apps] +- UDP protocol support for Agnhost connect subcommand ([#98639](https://github.com/kubernetes/kubernetes/pull/98639), [@knabben](https://github.com/knabben)) [SIG Testing] +- Upgrades `IPv6Dualstack` to `Beta` and turns it on by default. Clusters new and existing will not be affected until user starting adding secondary pod and service cidrs cli flags as described here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack ([#98969](https://github.com/kubernetes/kubernetes/pull/98969), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, Cloud Provider, Network and Node] ### Documentation -- Kubelet: remove alpha warnings for CNI flags. ([#94508](https://github.com/kubernetes/kubernetes/pull/94508), [@andrewsykim](https://github.com/andrewsykim)) [SIG Network and Node] +- Fix ALPHA stability level reference link ([#98641](https://github.com/kubernetes/kubernetes/pull/98641), [@Jeffwan](https://github.com/Jeffwan)) [SIG Auth, Cloud Provider, Instrumentation and Storage] ### Failing Test -- Kube-proxy iptables min-sync-period defaults to 1 sec. Previously, it was 0. ([#92836](https://github.com/kubernetes/kubernetes/pull/92836), [@aojea](https://github.com/aojea)) [SIG Network] +- Escape the special characters like `[`, `]` and ` ` that exist in vsphere windows path ([#98830](https://github.com/kubernetes/kubernetes/pull/98830), [@liyanhui1228](https://github.com/liyanhui1228)) [SIG Storage and Windows] +- Kube-proxy: fix a bug on UDP NodePort Services where stale conntrack entries may blackhole the traffic directed to the NodePort. ([#98305](https://github.com/kubernetes/kubernetes/pull/98305), [@aojea](https://github.com/aojea)) [SIG Network] ### Bug or Regression -- A panic in the apiserver caused by the `informer-sync` health checker is now fixed. ([#93600](https://github.com/kubernetes/kubernetes/pull/93600), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG API Machinery] -- Add kubectl wait --ignore-not-found flag ([#90969](https://github.com/kubernetes/kubernetes/pull/90969), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] -- Adding fix to the statefulset controller to wait for pvc deletion before creating pods. ([#93457](https://github.com/kubernetes/kubernetes/pull/93457), [@ymmt2005](https://github.com/ymmt2005)) [SIG Apps] -- Azure ARM client: don't segfault on empty response and http error ([#94078](https://github.com/kubernetes/kubernetes/pull/94078), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] -- Azure: fix a bug that kube-controller-manager would panic if wrong Azure VMSS name is configured ([#94306](https://github.com/kubernetes/kubernetes/pull/94306), [@knight42](https://github.com/knight42)) [SIG Cloud Provider] -- Azure: per VMSS VMSS VMs cache to prevent throttling on clusters having many attached VMSS ([#93107](https://github.com/kubernetes/kubernetes/pull/93107), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] -- Both apiserver_request_duration_seconds metrics and RequestReceivedTimestamp field of an audit event take - into account the time a request spends in the apiserver request filters. ([#94903](https://github.com/kubernetes/kubernetes/pull/94903), [@tkashem](https://github.com/tkashem)) [SIG API Machinery, Auth and Instrumentation] -- Build/lib/release: Explicitly use '--platform' in building server images - - When we switched to go-runner for building the apiserver, - controller-manager, and scheduler server components, we no longer - reference the individual architectures in the image names, specifically - in the 'FROM' directive of the server image Dockerfiles. - - As a result, server images for non-amd64 images copy in the go-runner - amd64 binary instead of the go-runner that matches that architecture. - - This commit explicitly sets the '--platform=linux/${arch}' to ensure - we're pulling the correct go-runner arch from the manifest list. - - Before: - `FROM ${base_image}` - - After: - `FROM --platform=linux/${arch} ${base_image}` ([#94552](https://github.com/kubernetes/kubernetes/pull/94552), [@justaugustus](https://github.com/justaugustus)) [SIG Release] -- CSIDriver object can be deployed during volume attachment. ([#93710](https://github.com/kubernetes/kubernetes/pull/93710), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Node, Storage and Testing] -- CVE-2020-8557 (Medium): Node-local denial of service via container /etc/hosts file. See https://github.com/kubernetes/kubernetes/issues/93032 for more details. ([#92916](https://github.com/kubernetes/kubernetes/pull/92916), [@joelsmith](https://github.com/joelsmith)) [SIG Node] -- Do not add nodes labeled with kubernetes.azure.com/managed=false to backend pool of load balancer. ([#93034](https://github.com/kubernetes/kubernetes/pull/93034), [@matthias50](https://github.com/matthias50)) [SIG Cloud Provider] -- Do not fail sorting empty elements. ([#94666](https://github.com/kubernetes/kubernetes/pull/94666), [@soltysh](https://github.com/soltysh)) [SIG CLI] -- Do not retry volume expansion if CSI driver returns FailedPrecondition error ([#92986](https://github.com/kubernetes/kubernetes/pull/92986), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage] -- Dockershim security: pod sandbox now always run with `no-new-privileges` and `runtime/default` seccomp profile - dockershim seccomp: custom profiles can now have smaller seccomp profiles when set at pod level ([#90948](https://github.com/kubernetes/kubernetes/pull/90948), [@pjbgf](https://github.com/pjbgf)) [SIG Node] -- Dual-stack: make nodeipam compatible with existing single-stack clusters when dual-stack feature gate become enabled by default ([#90439](https://github.com/kubernetes/kubernetes/pull/90439), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery] -- Endpoint controller requeues service after an endpoint deletion event occurs to confirm that deleted endpoints are undesired to mitigate the effects of an out of sync endpoint cache. ([#93030](https://github.com/kubernetes/kubernetes/pull/93030), [@swetharepakula](https://github.com/swetharepakula)) [SIG Apps and Network] -- EndpointSlice controllers now return immediately if they encounter an error creating, updating, or deleting resources. ([#93908](https://github.com/kubernetes/kubernetes/pull/93908), [@robscott](https://github.com/robscott)) [SIG Apps and Network] -- EndpointSliceMirroring controller now copies labels from Endpoints to EndpointSlices. ([#93442](https://github.com/kubernetes/kubernetes/pull/93442), [@robscott](https://github.com/robscott)) [SIG Apps and Network] -- EndpointSliceMirroring controller now mirrors Endpoints that do not have a Service associated with them. ([#94171](https://github.com/kubernetes/kubernetes/pull/94171), [@robscott](https://github.com/robscott)) [SIG Apps, Network and Testing] -- Ensure backoff step is set to 1 for Azure armclient. ([#94180](https://github.com/kubernetes/kubernetes/pull/94180), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] -- Ensure getPrimaryInterfaceID not panic when network interfaces for Azure VMSS are null ([#94355](https://github.com/kubernetes/kubernetes/pull/94355), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] -- Eviction requests for pods that have a non-zero DeletionTimestamp will always succeed ([#91342](https://github.com/kubernetes/kubernetes/pull/91342), [@michaelgugino](https://github.com/michaelgugino)) [SIG Apps] -- Extended DSR loadbalancer feature in winkernel kube-proxy to HNS versions 9.3-9.max, 10.2+ ([#93080](https://github.com/kubernetes/kubernetes/pull/93080), [@elweb9858](https://github.com/elweb9858)) [SIG Network] -- Fix HandleCrash order ([#93108](https://github.com/kubernetes/kubernetes/pull/93108), [@lixiaobing1](https://github.com/lixiaobing1)) [SIG API Machinery] -- Fix a concurrent map writes error in kubelet ([#93773](https://github.com/kubernetes/kubernetes/pull/93773), [@knight42](https://github.com/knight42)) [SIG Node] -- Fix a regression where kubeadm bails out with a fatal error when an optional version command line argument is supplied to the "kubeadm upgrade plan" command ([#94421](https://github.com/kubernetes/kubernetes/pull/94421), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] -- Fix azure file migration panic ([#94853](https://github.com/kubernetes/kubernetes/pull/94853), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix bug where loadbalancer deletion gets stuck because of missing resource group #75198 ([#93962](https://github.com/kubernetes/kubernetes/pull/93962), [@phiphi282](https://github.com/phiphi282)) [SIG Cloud Provider] -- Fix calling AttachDisk on a previously attached EBS volume ([#93567](https://github.com/kubernetes/kubernetes/pull/93567), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider, Storage and Testing] -- Fix detection of image filesystem, disk metrics for devicemapper, detection of OOM Kills on 5.0+ linux kernels. ([#92919](https://github.com/kubernetes/kubernetes/pull/92919), [@dashpole](https://github.com/dashpole)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node] -- Fix etcd_object_counts metric reported by kube-apiserver ([#94773](https://github.com/kubernetes/kubernetes/pull/94773), [@tkashem](https://github.com/tkashem)) [SIG API Machinery] -- Fix incorrectly reported verbs for kube-apiserver metrics for CRD objects ([#93523](https://github.com/kubernetes/kubernetes/pull/93523), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] -- Fix instance not found issues when an Azure Node is recreated in a short time ([#93316](https://github.com/kubernetes/kubernetes/pull/93316), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] -- Fix kube-apiserver /readyz to contain "informer-sync" check ensuring that internal informers are synced. ([#93670](https://github.com/kubernetes/kubernetes/pull/93670), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] -- Fix kubectl SchemaError on CRDs with schema using x-kubernetes-preserve-unknown-fields on array types. ([#94888](https://github.com/kubernetes/kubernetes/pull/94888), [@sttts](https://github.com/sttts)) [SIG API Machinery] -- Fix memory leak in EndpointSliceTracker for EndpointSliceMirroring controller. ([#93441](https://github.com/kubernetes/kubernetes/pull/93441), [@robscott](https://github.com/robscott)) [SIG Apps and Network] -- Fix missing csi annotations on node during parallel csinode update. ([#94389](https://github.com/kubernetes/kubernetes/pull/94389), [@pacoxu](https://github.com/pacoxu)) [SIG Storage] -- Fix the `cloudprovider_azure_api_request_duration_seconds` metric buckets to correctly capture the latency metrics. Previously, the majority of the calls would fall in the "+Inf" bucket. ([#94873](https://github.com/kubernetes/kubernetes/pull/94873), [@marwanad](https://github.com/marwanad)) [SIG Cloud Provider and Instrumentation] -- Fix: azure disk resize error if source does not exist ([#93011](https://github.com/kubernetes/kubernetes/pull/93011), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix: detach azure disk broken on Azure Stack ([#94885](https://github.com/kubernetes/kubernetes/pull/94885), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fix: determine the correct ip config based on ip family ([#93043](https://github.com/kubernetes/kubernetes/pull/93043), [@aramase](https://github.com/aramase)) [SIG Cloud Provider] -- Fix: initial delay in mounting azure disk & file ([#93052](https://github.com/kubernetes/kubernetes/pull/93052), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Fix: use sensitiveOptions on Windows mount ([#94126](https://github.com/kubernetes/kubernetes/pull/94126), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] -- Fixed Ceph RBD volume expansion when no ceph.conf exists ([#92027](https://github.com/kubernetes/kubernetes/pull/92027), [@juliantaylor](https://github.com/juliantaylor)) [SIG Storage] -- Fixed a bug where improper storage and comparison of endpoints led to excessive API traffic from the endpoints controller ([#94112](https://github.com/kubernetes/kubernetes/pull/94112), [@damemi](https://github.com/damemi)) [SIG Apps, Network and Testing] -- Fixed a bug whereby the allocation of reusable CPUs and devices was not being honored when the TopologyManager was enabled ([#93189](https://github.com/kubernetes/kubernetes/pull/93189), [@klueska](https://github.com/klueska)) [SIG Node] -- Fixed a panic in kubectl debug when pod has multiple init containers or ephemeral containers ([#94580](https://github.com/kubernetes/kubernetes/pull/94580), [@kiyoshim55](https://github.com/kiyoshim55)) [SIG CLI] -- Fixed a regression that sometimes prevented `kubectl portforward` to work when TCP and UDP services were configured on the same port ([#94728](https://github.com/kubernetes/kubernetes/pull/94728), [@amorenoz](https://github.com/amorenoz)) [SIG CLI] -- Fixed bug in reflector that couldn't recover from "Too large resource version" errors with API servers 1.17.0-1.18.5 ([#94316](https://github.com/kubernetes/kubernetes/pull/94316), [@janeczku](https://github.com/janeczku)) [SIG API Machinery] -- Fixed bug where kubectl top pod output is not sorted when --sort-by and --containers flags are used together ([#93692](https://github.com/kubernetes/kubernetes/pull/93692), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] -- Fixed kubelet creating extra sandbox for pods with RestartPolicyOnFailure after all containers succeeded ([#92614](https://github.com/kubernetes/kubernetes/pull/92614), [@tnqn](https://github.com/tnqn)) [SIG Node and Testing] -- Fixed memory leak in endpointSliceTracker ([#92838](https://github.com/kubernetes/kubernetes/pull/92838), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network] -- Fixed node data lost in kube-scheduler for clusters with imbalance on number of nodes across zones ([#93355](https://github.com/kubernetes/kubernetes/pull/93355), [@maelk](https://github.com/maelk)) [SIG Scheduling] -- Fixed the EndpointSliceController to correctly create endpoints for IPv6-only pods. - - Fixed the EndpointController to allow IPv6 headless services, if the IPv6DualStack - feature gate is enabled, by specifying `ipFamily: IPv6` on the service. (This already - worked with the EndpointSliceController.) ([#91399](https://github.com/kubernetes/kubernetes/pull/91399), [@danwinship](https://github.com/danwinship)) [SIG Apps and Network] -- Fixes a bug evicting pods after a taint with a limited tolerationSeconds toleration is removed from a node ([#93722](https://github.com/kubernetes/kubernetes/pull/93722), [@liggitt](https://github.com/liggitt)) [SIG Apps and Node] -- Fixes a bug where EndpointSlices would not be recreated after rapid Service recreation. ([#94730](https://github.com/kubernetes/kubernetes/pull/94730), [@robscott](https://github.com/robscott)) [SIG Apps, Network and Testing] -- Fixes a race condition in kubelet pod handling ([#94751](https://github.com/kubernetes/kubernetes/pull/94751), [@auxten](https://github.com/auxten)) [SIG Node] -- Fixes an issue proxying to ipv6 pods without specifying a port ([#94834](https://github.com/kubernetes/kubernetes/pull/94834), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Network] -- Fixes an issue that can result in namespaced custom resources being orphaned when their namespace is deleted, if the CRD defining the custom resource is removed concurrently with namespaces being deleted, then recreated. ([#93790](https://github.com/kubernetes/kubernetes/pull/93790), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps] -- Ignore root user check when windows pod starts ([#92355](https://github.com/kubernetes/kubernetes/pull/92355), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] -- Increased maximum IOPS of AWS EBS io1 volumes to 64,000 (current AWS maximum). ([#90014](https://github.com/kubernetes/kubernetes/pull/90014), [@jacobmarble](https://github.com/jacobmarble)) [SIG Cloud Provider and Storage] -- K8s.io/apimachinery: runtime.DefaultUnstructuredConverter.FromUnstructured now handles converting integer fields to typed float values ([#93250](https://github.com/kubernetes/kubernetes/pull/93250), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] -- Kube-aggregator certificates are dynamically loaded on change from disk ([#92791](https://github.com/kubernetes/kubernetes/pull/92791), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] -- Kube-apiserver: fixed a bug returning inconsistent results from list requests which set a field or label selector and set a paging limit ([#94002](https://github.com/kubernetes/kubernetes/pull/94002), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] -- Kube-apiserver: jsonpath expressions with consecutive recursive descent operators are no longer evaluated for custom resource printer columns ([#93408](https://github.com/kubernetes/kubernetes/pull/93408), [@joelsmith](https://github.com/joelsmith)) [SIG API Machinery] -- Kube-proxy now trims extra spaces found in loadBalancerSourceRanges to match Service validation. ([#94107](https://github.com/kubernetes/kubernetes/pull/94107), [@robscott](https://github.com/robscott)) [SIG Network] -- Kube-up now includes CoreDNS version v1.7.0. Some of the major changes include: - - Fixed a bug that could cause CoreDNS to stop updating service records. - - Fixed a bug in the forward plugin where only the first upstream server is always selected no matter which policy is set. - - Remove already deprecated options `resyncperiod` and `upstream` in the Kubernetes plugin. - - Includes Prometheus metrics name changes (to bring them in line with standard Prometheus metrics naming convention). They will be backward incompatible with existing reporting formulas that use the old metrics' names. - - The federation plugin (allows for v1 Kubernetes federation) has been removed. - More details are available in https://coredns.io/2020/06/15/coredns-1.7.0-release/ ([#92718](https://github.com/kubernetes/kubernetes/pull/92718), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cloud Provider] -- Kubeadm now makes sure the etcd manifest is regenerated upon upgrade even when no etcd version change takes place ([#94395](https://github.com/kubernetes/kubernetes/pull/94395), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] -- Kubeadm: avoid a panic when determining if the running version of CoreDNS is supported during upgrades ([#94299](https://github.com/kubernetes/kubernetes/pull/94299), [@zouyee](https://github.com/zouyee)) [SIG Cluster Lifecycle] -- Kubeadm: ensure "kubeadm reset" does not unmount the root "/var/lib/kubelet" directory if it is mounted by the user ([#93702](https://github.com/kubernetes/kubernetes/pull/93702), [@thtanaka](https://github.com/thtanaka)) [SIG Cluster Lifecycle] -- Kubeadm: ensure the etcd data directory is created with 0700 permissions during control-plane init and join ([#94102](https://github.com/kubernetes/kubernetes/pull/94102), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: fix the bug that kubeadm tries to call 'docker info' even if the CRI socket was for another CR ([#94555](https://github.com/kubernetes/kubernetes/pull/94555), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] -- Kubeadm: make the kubeconfig files for the kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. This makes kubeadm clusters more reseliant to version skew problems during immutable upgrades: https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-controller-manager-kube-scheduler-and-cloud-controller-manager ([#94398](https://github.com/kubernetes/kubernetes/pull/94398), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: relax the validation of kubeconfig server URLs. Allow the user to define custom kubeconfig server URLs without erroring out during validation of existing kubeconfig files (e.g. when using external CA mode). ([#94816](https://github.com/kubernetes/kubernetes/pull/94816), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubeadm: remove duplicate DNS names and IP addresses from generated certificates ([#92753](https://github.com/kubernetes/kubernetes/pull/92753), [@QianChenglong](https://github.com/QianChenglong)) [SIG Cluster Lifecycle] -- Kubelet: assume that swap is disabled when `/proc/swaps` does not exist ([#93931](https://github.com/kubernetes/kubernetes/pull/93931), [@SataQiu](https://github.com/SataQiu)) [SIG Node] -- Kubelet: fix race condition in pluginWatcher ([#93622](https://github.com/kubernetes/kubernetes/pull/93622), [@knight42](https://github.com/knight42)) [SIG Node] -- Kuberuntime security: pod sandbox now always runs with `runtime/default` seccomp profile - kuberuntime seccomp: custom profiles can now have smaller seccomp profiles when set at pod level ([#90949](https://github.com/kubernetes/kubernetes/pull/90949), [@pjbgf](https://github.com/pjbgf)) [SIG Node] -- NONE ([#71269](https://github.com/kubernetes/kubernetes/pull/71269), [@DeliangFan](https://github.com/DeliangFan)) [SIG Node] -- New Azure instance types do now have correct max data disk count information. ([#94340](https://github.com/kubernetes/kubernetes/pull/94340), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Cloud Provider and Storage] -- Pods with invalid Affinity/AntiAffinity LabelSelectors will now fail scheduling when these plugins are enabled ([#93660](https://github.com/kubernetes/kubernetes/pull/93660), [@damemi](https://github.com/damemi)) [SIG Scheduling] -- Require feature flag CustomCPUCFSQuotaPeriod if setting a non-default cpuCFSQuotaPeriod in kubelet config. ([#94687](https://github.com/kubernetes/kubernetes/pull/94687), [@karan](https://github.com/karan)) [SIG Node] -- Reverted devicemanager for Windows node added in 1.19rc1. ([#93263](https://github.com/kubernetes/kubernetes/pull/93263), [@liggitt](https://github.com/liggitt)) [SIG Node and Windows] -- Scheduler bugfix: Scheduler doesn't lose pod information when nodes are quickly recreated. This could happen when nodes are restarted or quickly recreated reusing a nodename. ([#93938](https://github.com/kubernetes/kubernetes/pull/93938), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scalability, Scheduling and Testing] -- The EndpointSlice controller now waits for EndpointSlice and Node caches to be synced before starting. ([#94086](https://github.com/kubernetes/kubernetes/pull/94086), [@robscott](https://github.com/robscott)) [SIG Apps and Network] -- The `/debug/api_priority_and_fairness/dump_requests` path at an apiserver will no longer return a phantom line for each exempt priority level. ([#93406](https://github.com/kubernetes/kubernetes/pull/93406), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] -- The kubelet recognizes the --containerd-namespace flag to configure the namespace used by cadvisor. ([#87054](https://github.com/kubernetes/kubernetes/pull/87054), [@changyaowei](https://github.com/changyaowei)) [SIG Node] -- The terminationGracePeriodSeconds from pod spec is respected for the mirror pod. ([#92442](https://github.com/kubernetes/kubernetes/pull/92442), [@tedyu](https://github.com/tedyu)) [SIG Node and Testing] -- Update Calico to v3.15.2 ([#94241](https://github.com/kubernetes/kubernetes/pull/94241), [@lmm](https://github.com/lmm)) [SIG Cloud Provider] -- Update default etcd server version to 3.4.13 ([#94287](https://github.com/kubernetes/kubernetes/pull/94287), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing] -- Updated Cluster Autoscaler to 1.19.0; ([#93577](https://github.com/kubernetes/kubernetes/pull/93577), [@vivekbagade](https://github.com/vivekbagade)) [SIG Autoscaling and Cloud Provider] -- Use NLB Subnet CIDRs instead of VPC CIDRs in Health Check SG Rules ([#93515](https://github.com/kubernetes/kubernetes/pull/93515), [@t0rr3sp3dr0](https://github.com/t0rr3sp3dr0)) [SIG Cloud Provider] -- Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. ([#92817](https://github.com/kubernetes/kubernetes/pull/92817), [@kmala](https://github.com/kmala)) [SIG Node] -- Very large patches may now be specified to `kubectl patch` with the `--patch-file` flag instead of including them directly on the command line. The `--patch` and `--patch-file` flags are mutually exclusive. ([#93548](https://github.com/kubernetes/kubernetes/pull/93548), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] -- When creating a networking.k8s.io/v1 Ingress API object, `spec.rules[*].http` values are now validated consistently when the `host` field contains a wildcard. ([#93954](https://github.com/kubernetes/kubernetes/pull/93954), [@Miciah](https://github.com/Miciah)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing] +- Add missing --kube-api-content-type in kubemark hollow template ([#98911](https://github.com/kubernetes/kubernetes/pull/98911), [@Jeffwan](https://github.com/Jeffwan)) [SIG Scalability and Testing] +- Avoid duplicate error messages when runing kubectl edit quota ([#98201](https://github.com/kubernetes/kubernetes/pull/98201), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery and Apps] +- Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. ([#98133](https://github.com/kubernetes/kubernetes/pull/98133), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Fix errors when accessing Windows container stats for Dockershim ([#98510](https://github.com/kubernetes/kubernetes/pull/98510), [@jsturtevant](https://github.com/jsturtevant)) [SIG Node and Windows] +- Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. ([#99127](https://github.com/kubernetes/kubernetes/pull/99127), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] +- In the method that ensures that the docker and containerd are in the correct containers with the proper OOM score set up, fixed the bug of identifying containerd process. ([#97888](https://github.com/kubernetes/kubernetes/pull/97888), [@pacoxu](https://github.com/pacoxu)) [SIG Node] +- Kubelet now cleans up orphaned volume directories automatically ([#95301](https://github.com/kubernetes/kubernetes/pull/95301), [@lorenz](https://github.com/lorenz)) [SIG Node and Storage] +- When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. ([#99122](https://github.com/kubernetes/kubernetes/pull/99122), [@huffmanca](https://github.com/huffmanca)) [SIG Cloud Provider and Storage] ### Other (Cleanup or Flake) -- --cache-dir sets cache directory for both http and discovery, defaults to $HOME/.kube/cache ([#92910](https://github.com/kubernetes/kubernetes/pull/92910), [@soltysh](https://github.com/soltysh)) [SIG API Machinery and CLI] -- Adds a bootstrapping ClusterRole, ClusterRoleBinding and group for /metrics, /livez/*, /readyz/*, & /healthz/- endpoints. ([#93311](https://github.com/kubernetes/kubernetes/pull/93311), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Auth, Cloud Provider and Instrumentation] -- Base-images: Update to debian-iptables:buster-v1.3.0 +- APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now available under k8s.io/kubelet/pkg/apis/ ([#98931](https://github.com/kubernetes/kubernetes/pull/98931), [@michaelbeaumont](https://github.com/michaelbeaumont)) [SIG Apps, Auth and Node] +- Migrate `pkg/kubelet/(pod, pleg)` to structured logging ([#98990](https://github.com/kubernetes/kubernetes/pull/98990), [@gjkim42](https://github.com/gjkim42)) [SIG Instrumentation and Node] +- Migrate pkg/kubelet/nodestatus to structured logging ([#99001](https://github.com/kubernetes/kubernetes/pull/99001), [@QiWang19](https://github.com/QiWang19)) [SIG Node] +- Migrate pkg/kubelet/server logs to structured logging ([#98643](https://github.com/kubernetes/kubernetes/pull/98643), [@chenyw1990](https://github.com/chenyw1990)) [SIG Node] +- Migrate proxy/winkernel/proxier.go logs to structured logging ([#98001](https://github.com/kubernetes/kubernetes/pull/98001), [@JornShen](https://github.com/JornShen)) [SIG Network and Windows] +- Migrate scheduling_queue.go to structured logging ([#98358](https://github.com/kubernetes/kubernetes/pull/98358), [@tanjing2020](https://github.com/tanjing2020)) [SIG Scheduling] +- Several flags related to the deprecated dockershim which are present in the kubelet command line are now deprecated. ([#98730](https://github.com/kubernetes/kubernetes/pull/98730), [@dims](https://github.com/dims)) [SIG Node] +- The deprecated feature gates `CSIDriverRegistry`, `BlockVolume` and `CSIBlockVolume` are now unconditionally enabled and can no longer be specified in component invocations. ([#98021](https://github.com/kubernetes/kubernetes/pull/98021), [@gavinfish](https://github.com/gavinfish)) [SIG Storage] + +## Dependencies + +### Added +_Nothing has changed._ + +### Changed +- sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.0.3 + +### Removed +_Nothing has changed._ + + + +# v1.21.0-alpha.3 + + +## Downloads for v1.21.0-alpha.3 + +### Source Code + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes.tar.gz) | 704ec916a1dbd134c54184d2652671f80ae09274f9d23dbbed312944ebeccbc173e2e6b6949b38bdbbfdaf8aa032844deead5efeda1b3150f9751386d9184bc8 +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-src.tar.gz) | 57db9e7560cfc9c10e7059cb5faf9c4bd5eb8f9b7964f44f000a417021cf80873184b774e7c66c80d4aba84c14080c6bc335618db3d2e5f276436ae065e25408 + +### Client binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | e2706efda92d5cf4f8b69503bb2f7703a8754407eff7f199bb77847838070e720e5f572126c14daa4c0c03b59bb1a63c1dfdeb6e936a40eff1d5497e871e3409 +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-386.tar.gz) | 007bb23c576356ed0890bdfd25a0f98d552599e0ffec19fb982591183c7c1f216d8a3ffa3abf15216be12ae5c4b91fdcd48a7306a2d26b007b86a6abd553fc61 +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | 39504b0c610348beba60e8866fff265bad58034f74504951cd894c151a248db718d10f77ebc83f2c38b2d517f8513a46325b38889eefa261ca6dbffeceba50ff +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | 30bc2c40d0c759365422ad1651a6fb35909be771f463c5b971caf401f9209525d05256ab70c807e88628dd357c2896745eecf13eda0b748464da97d0a5ef2066 +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | 085cdf574dc8fd33ece667130b8c45830b522a07860e03a2384283b1adea73a9652ef3dfaa566e69ee00aea1a6461608814b3ce7a3f703e4a934304f7ae12f97 +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | b34b845037d83ea7b3e2d80a9ede4f889b71b17b93b1445f0d936a36e98c13ed6ada125630a68d9243a5fcd311ee37cdcc0c05da484da8488ea5060bc529dbfc +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | c4758adc7a404b776556efaa79655db2a70777c562145d6ea6887f3335988367a0c2fcd4383e469340f2a768b22e786951de212805ca1cb91104d41c21e0c9ce +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-windows-386.tar.gz) | f51edc79702bbd1d9cb3a672852a405e11b20feeab64c5411a7e85c9af304960663eb6b23ef96e0f8c44a722fecf58cb6d700ea2c42c05b3269d8efd5ad803f2 +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | 6a3507ce4ac40a0dc7e4720538863fa15f8faf025085a032f34b8fa0f6fa4e8c26849baf649b5b32829b9182e04f82721b13950d31cf218c35be6bf1c05d6abf + +### Server binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | 19181d162dfb0b30236e2bf1111000e037eece87c037ca2b24622ca94cb88db86aa4da4ca533522518b209bc9983bbfd6b880a7898e0da96b33f3f6c4690539b +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-arm.tar.gz) | 42a02f9e08a78ad5da6e5fa1ab12bf1e3c967c472fdbdadbd8746586da74dc8093682ba8513ff2a5301393c47ee9021b860e88ada56b13da386ef485708e46ca +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | 3c8ba8eb02f70061689bd7fab7813542005efe2edc6cfc6b7aecd03ffedf0b81819ad91d69fff588e83023d595eefbfe636aa55e1856add8733bf42fff3c748f +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | cd9e6537450411c39a06fd0b5819db3d16b668d403fb3627ec32c0e32dd1c4860e942934578ca0e1d1b8e6f21f450ff81e37e0cd46ff5c5faf7847ab074aefc5 +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | ada3f65e53bc0e0c0229694dd48c425388089d6d77111a62476d1b08f6ad1d8ab3d60b9ed7d95ac1b42c2c6be8dc0618f40679717160769743c43583d8452362 + +### Node binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | ae0fec6aa59e49624b55d9a11c12fdf717ddfe04bdfd4f69965d03004a34e52ee4a3e83f7b61d0c6a86f43b72c99f3decb195b39ae529ef30526d18ec5f58f83 +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-arm.tar.gz) | 9a48c140ab53b7ed8ecec6903988a1a474efc16d2538e5974bc9a12f0c9190be78c4f9e326bf4e982d0b7045a80b99dd0fda7e9b650663be5b89bfd991596746 +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | 6912adbc9300344bea470d6435f7b387bfce59767078c11728ce59faf47cd3f72b41b9604fcc5cda45e9816fe939fbe2fb33e52a773e6ff2dfa9a615b4df6141 +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | d66dccfe3e6ed6d81567c70703f15375a53992b3a5e2814b98c32e581b861ad95912e03ed2562415d087624c008038bb4a816611fa255442ae752968ea15856b +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | ad8c69a28f1fbafa3f1cb54909bfd3fc22b104bed63d7ca2b296208c9d43eb5f2943a0ff267da4c185186cdd9f7f77b315cd7f5f1bf9858c0bf42eceb9ac3c58 +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | 91d723aa848a9cb028f5bcb41090ca346fb973961521d025c4399164de2c8029b57ca2c4daca560d3c782c05265d2eb0edb0abcce6f23d3efbecf2316a54d650 + +## Changelog since v1.21.0-alpha.2 + +## Urgent Upgrade Notes + +### (No, really, you MUST read this before you upgrade) + + - Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. ([#98700](https://github.com/kubernetes/kubernetes/pull/98700), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider, Storage and Testing] + - Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in 1.22. ([#98243](https://github.com/kubernetes/kubernetes/pull/98243), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Node and Storage] + +## Changes by Kind + +### API Change + +- Adds support for portRange / EndPort in Network Policy ([#97058](https://github.com/kubernetes/kubernetes/pull/97058), [@rikatz](https://github.com/rikatz)) [SIG Apps and Network] +- Fixes using server-side apply with APIService resources ([#98576](https://github.com/kubernetes/kubernetes/pull/98576), [@kevindelgado](https://github.com/kevindelgado)) [SIG API Machinery, Apps and Testing] +- Kubernetes is now built using go1.15.7 ([#98363](https://github.com/kubernetes/kubernetes/pull/98363), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Node, Release and Testing] +- Scheduler extender filter interface now can report unresolvable failed nodes in the new field `FailedAndUnresolvableNodes` of `ExtenderFilterResult` struct. Nodes in this map will be skipped in the preemption phase. ([#92866](https://github.com/kubernetes/kubernetes/pull/92866), [@cofyc](https://github.com/cofyc)) [SIG Scheduling] + +### Feature + +- A lease can only attach up to 10k objects. ([#98257](https://github.com/kubernetes/kubernetes/pull/98257), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery] +- Add ignore-errors flag for drain, support none-break drain in group ([#98203](https://github.com/kubernetes/kubernetes/pull/98203), [@yuzhiquan](https://github.com/yuzhiquan)) [SIG CLI] +- Base-images: Update to debian-iptables:buster-v1.4.0 - Uses iptables 1.8.5 - - base-images: Update to debian-base:buster-v1.2.0 - - cluster/images/etcd: Build etcd:3.4.13-1 image - - Uses debian-base:buster-v1.2.0 ([#94733](https://github.com/kubernetes/kubernetes/pull/94733), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release and Testing] -- Build: Update to debian-base@v2.1.2 and debian-iptables@v12.1.1 ([#93667](https://github.com/kubernetes/kubernetes/pull/93667), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release and Testing] -- Build: Update to debian-base@v2.1.3 and debian-iptables@v12.1.2 ([#93916](https://github.com/kubernetes/kubernetes/pull/93916), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release and Testing] -- Build: Update to go-runner:buster-v2.0.0 ([#94167](https://github.com/kubernetes/kubernetes/pull/94167), [@justaugustus](https://github.com/justaugustus)) [SIG Release] -- Fix kubelet to properly log when a container is started. Before, sometimes the log said that a container is dead and was restarted when it was started for the first time. This only happened when using pods with initContainers and regular containers. ([#91469](https://github.com/kubernetes/kubernetes/pull/91469), [@rata](https://github.com/rata)) [SIG Node] -- Fix: license issue in blob disk feature ([#92824](https://github.com/kubernetes/kubernetes/pull/92824), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] -- Fixes the flooding warning messages about setting volume ownership for configmap/secret volumes ([#92878](https://github.com/kubernetes/kubernetes/pull/92878), [@jvanz](https://github.com/jvanz)) [SIG Instrumentation, Node and Storage] -- Fixes the message about no auth for metrics in scheduler. ([#94035](https://github.com/kubernetes/kubernetes/pull/94035), [@zhouya0](https://github.com/zhouya0)) [SIG Scheduling] -- Kube-up: defaults to limiting critical pods to the kube-system namespace to match behavior prior to 1.17 ([#93121](https://github.com/kubernetes/kubernetes/pull/93121), [@liggitt](https://github.com/liggitt)) [SIG Cloud Provider and Scheduling] -- Kubeadm: Separate argument key/value in log msg ([#94016](https://github.com/kubernetes/kubernetes/pull/94016), [@mrueg](https://github.com/mrueg)) [SIG Cluster Lifecycle] -- Kubeadm: remove support for the "ci/k8s-master" version label. This label has been removed in the Kubernetes CI release process and would no longer work in kubeadm. You can use the "ci/latest" version label instead. See kubernetes/test-infra#18517 ([#93626](https://github.com/kubernetes/kubernetes/pull/93626), [@vikkyomkar](https://github.com/vikkyomkar)) [SIG Cluster Lifecycle] -- Kubeadm: remove the CoreDNS check for known image digests when applying the addon ([#94506](https://github.com/kubernetes/kubernetes/pull/94506), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] -- Kubernetes is now built with go1.15.0 ([#93939](https://github.com/kubernetes/kubernetes/pull/93939), [@justaugustus](https://github.com/justaugustus)) [SIG Release and Testing] -- Kubernetes is now built with go1.15.0-rc.2 ([#93827](https://github.com/kubernetes/kubernetes/pull/93827), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing] -- Lock ExternalPolicyForExternalIP to default, this feature gate will be removed in 1.22. ([#94581](https://github.com/kubernetes/kubernetes/pull/94581), [@knabben](https://github.com/knabben)) [SIG Network] -- Service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset is removed. All Standard load balancers will always enable tcp resets. ([#94297](https://github.com/kubernetes/kubernetes/pull/94297), [@MarcPow](https://github.com/MarcPow)) [SIG Cloud Provider] -- Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver ([#94397](https://github.com/kubernetes/kubernetes/pull/94397), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] -- Strip unnecessary security contexts on Windows ([#93475](https://github.com/kubernetes/kubernetes/pull/93475), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Node, Testing and Windows] -- To ensure the code be strong, add unit test for GetAddressAndDialer ([#93180](https://github.com/kubernetes/kubernetes/pull/93180), [@FreeZhang61](https://github.com/FreeZhang61)) [SIG Node] -- Update CNI plugins to v0.8.7 ([#94367](https://github.com/kubernetes/kubernetes/pull/94367), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Network, Node, Release and Testing] -- Update Golang to v1.14.5 - - Update repo-infra to 0.0.7 (to support go1.14.5 and go1.13.13) - - Includes: - - bazelbuild/bazel-toolchains@3.3.2 - - bazelbuild/rules_go@v0.22.7 ([#93088](https://github.com/kubernetes/kubernetes/pull/93088), [@justaugustus](https://github.com/justaugustus)) [SIG Release and Testing] -- Update Golang to v1.14.6 - - Update repo-infra to 0.0.8 (to support go1.14.6 and go1.13.14) - - Includes: - - bazelbuild/bazel-toolchains@3.4.0 - - bazelbuild/rules_go@v0.22.8 ([#93198](https://github.com/kubernetes/kubernetes/pull/93198), [@justaugustus](https://github.com/justaugustus)) [SIG Release and Testing] -- Update cri-tools to [v1.19.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.19.0) ([#94307](https://github.com/kubernetes/kubernetes/pull/94307), [@xmudrii](https://github.com/xmudrii)) [SIG Cloud Provider] -- Update default etcd server version to 3.4.9 ([#92349](https://github.com/kubernetes/kubernetes/pull/92349), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing] -- Update etcd client side to v3.4.13 ([#94259](https://github.com/kubernetes/kubernetes/pull/94259), [@jingyih](https://github.com/jingyih)) [SIG API Machinery and Cloud Provider] -- `kubectl get ingress` now prefers the `networking.k8s.io/v1` over `extensions/v1beta1` (deprecated since v1.14). To explicitly request the deprecated version, use `kubectl get ingress.v1beta1.extensions`. ([#94309](https://github.com/kubernetes/kubernetes/pull/94309), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and CLI] + - base-images: Update to debian-base:buster-v1.3.0 + - cluster/images/etcd: Build etcd:3.4.13-2 image + - Uses debian-base:buster-v1.3.0 ([#98401](https://github.com/kubernetes/kubernetes/pull/98401), [@pacoxu](https://github.com/pacoxu)) [SIG Testing] +- Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. ([#98324](https://github.com/kubernetes/kubernetes/pull/98324), [@atosatto](https://github.com/atosatto)) [SIG API Machinery] +- Kubectl wait ensures that observedGeneration >= generation if applicable ([#97408](https://github.com/kubernetes/kubernetes/pull/97408), [@KnicKnic](https://github.com/KnicKnic)) [SIG CLI] +- Kubernetes is now built using go1.15.8 ([#98834](https://github.com/kubernetes/kubernetes/pull/98834), [@cpanato](https://github.com/cpanato)) [SIG Cloud Provider, Instrumentation, Release and Testing] +- New admission controller "denyserviceexternalips" is available. Clusters which do not *need- the Service "externalIPs" feature should enable this controller and be more secure. ([#97395](https://github.com/kubernetes/kubernetes/pull/97395), [@thockin](https://github.com/thockin)) [SIG API Machinery] +- Overall, enable the feature of `PreferNominatedNode` will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of `PreferNominatedNode`, the pod might not be scheduled to the best candidate node in the cluster. ([#93179](https://github.com/kubernetes/kubernetes/pull/93179), [@chendave](https://github.com/chendave)) [SIG Scheduling and Testing] +- Pause image upgraded to 3.4.1 in kubelet and kubeadm for both Linux and Windows. ([#98205](https://github.com/kubernetes/kubernetes/pull/98205), [@pacoxu](https://github.com/pacoxu)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Testing and Windows] +- The `ServiceAccountIssuerDiscovery` feature has graduated to GA, and is unconditionally enabled. The `ServiceAccountIssuerDiscovery` feature-gate will be removed in 1.22. ([#98553](https://github.com/kubernetes/kubernetes/pull/98553), [@mtaufen](https://github.com/mtaufen)) [SIG API Machinery, Auth and Testing] + +### Documentation + +- Feat: azure file migration go beta in 1.21. Feature gates CSIMigration to Beta (on by default) and CSIMigrationAzureFile to Beta (off by default since it requires installation of the AzureFile CSI Driver) + The in-tree AzureFile plugin "kubernetes.io/azure-file" is now deprecated and will be removed in 1.23. Users should enable CSIMigration + CSIMigrationAzureFile features and install the AzureFile CSI Driver (https://github.com/kubernetes-sigs/azurefile-csi-driver) to avoid disruption to existing Pod and PVC objects at that time. + Users should start using the AzureFile CSI Driver directly for any new volumes. ([#96293](https://github.com/kubernetes/kubernetes/pull/96293), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] + +### Failing Test + +- Kubelet: the HostPort implementation in dockershim was not taking into consideration the HostIP field, causing that the same HostPort can not be used with different IP addresses. + This bug causes the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. ([#98755](https://github.com/kubernetes/kubernetes/pull/98755), [@aojea](https://github.com/aojea)) [SIG Cloud Provider, Network and Node] + +### Bug or Regression + +- Fix NPE in ephemeral storage eviction ([#98261](https://github.com/kubernetes/kubernetes/pull/98261), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. + Added iptables rules to allow healthcheck nodeport traffic. ([#97824](https://github.com/kubernetes/kubernetes/pull/97824), [@hanlins](https://github.com/hanlins)) [SIG Network] +- Fixed kube-proxy container image architecture for non amd64 images. ([#98526](https://github.com/kubernetes/kubernetes/pull/98526), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Release and Testing] +- Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. ([#98311](https://github.com/kubernetes/kubernetes/pull/98311), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] +- Fixes a panic in the disruption budget controller for PDB objects with invalid selectors ([#98750](https://github.com/kubernetes/kubernetes/pull/98750), [@mortent](https://github.com/mortent)) [SIG Apps] +- Fixes connection errors when using `--volume-host-cidr-denylist` or `--volume-host-allow-local-loopback` ([#98436](https://github.com/kubernetes/kubernetes/pull/98436), [@liggitt](https://github.com/liggitt)) [SIG Network and Storage] +- If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400. + - in cases where the client specifies a timeout in the request URL, the overall request deadline is shortened now since the deadline is setup as soon as the request is received by the apiserver. ([#96901](https://github.com/kubernetes/kubernetes/pull/96901), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing] +- Kubeadm: Some text in the `kubeadm upgrade plan` output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. ([#98728](https://github.com/kubernetes/kubernetes/pull/98728), [@stmcginnis](https://github.com/stmcginnis)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. ([#98882](https://github.com/kubernetes/kubernetes/pull/98882), [@kvaps](https://github.com/kvaps)) [SIG Cluster Lifecycle] +- Kubeadm: fix bad token placeholder text in "config print *-defaults --help" ([#98839](https://github.com/kubernetes/kubernetes/pull/98839), [@Mattias-](https://github.com/Mattias-)) [SIG Cluster Lifecycle] +- Kubeadm: get k8s CI version markers from k8s infra bucket ([#98836](https://github.com/kubernetes/kubernetes/pull/98836), [@hasheddan](https://github.com/hasheddan)) [SIG Cluster Lifecycle and Release] +- Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. ([#97934](https://github.com/kubernetes/kubernetes/pull/97934), [@mattcary](https://github.com/mattcary)) [SIG Cloud Provider and Storage] +- Remove CSI topology from migrated in-tree gcepd volume. ([#97823](https://github.com/kubernetes/kubernetes/pull/97823), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Cloud Provider and Storage] +- Sync node status during kubelet node shutdown. + Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. ([#98005](https://github.com/kubernetes/kubernetes/pull/98005), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. ([#98715](https://github.com/kubernetes/kubernetes/pull/98715), [@carlory](https://github.com/carlory)) [SIG Scheduling] +- We will no longer automatically delete all data when a failure is detected during creation of the volume data file on a CSI volume. Now we will only remove the data file and volume path. ([#96021](https://github.com/kubernetes/kubernetes/pull/96021), [@huffmanca](https://github.com/huffmanca)) [SIG Storage] + +### Other (Cleanup or Flake) + +- Fix the description of command line flags that can override --config ([#98254](https://github.com/kubernetes/kubernetes/pull/98254), [@changshuchao](https://github.com/changshuchao)) [SIG Scheduling] +- Migrate scheduler/taint_manager.go structured logging ([#98259](https://github.com/kubernetes/kubernetes/pull/98259), [@tanjing2020](https://github.com/tanjing2020)) [SIG Apps] +- Migrate staging/src/k8s.io/apiserver/pkg/admission logs to structured logging ([#98138](https://github.com/kubernetes/kubernetes/pull/98138), [@lala123912](https://github.com/lala123912)) [SIG API Machinery] +- Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object ([#98430](https://github.com/kubernetes/kubernetes/pull/98430), [@liggitt](https://github.com/liggitt)) [SIG Network and Testing] +- The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. ([#98325](https://github.com/kubernetes/kubernetes/pull/98325), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth, Cloud Provider and Scheduling] +- The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing `--allowed-not-ready-nodes=-1` when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. ([#98781](https://github.com/kubernetes/kubernetes/pull/98781), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] +- The feature gates `WindowsGMSA` and `WindowsRunAsUserName` that are GA since v1.18 are now removed. ([#96531](https://github.com/kubernetes/kubernetes/pull/96531), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Node and Windows] +- The new `-gce-zones` flag on the `e2e.test` binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. ([#98787](https://github.com/kubernetes/kubernetes/pull/98787), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Cluster Lifecycle and Testing] ## Dependencies ### Added -- github.com/Azure/go-autorest: [v14.2.0+incompatible](https://github.com/Azure/go-autorest/tree/v14.2.0) -- github.com/fvbommel/sortorder: [v1.0.1](https://github.com/fvbommel/sortorder/tree/v1.0.1) -- github.com/yuin/goldmark: [v1.1.27](https://github.com/yuin/goldmark/tree/v1.1.27) -- sigs.k8s.io/structured-merge-diff/v4: v4.0.1 +- github.com/moby/spdystream: [v0.2.0](https://github.com/moby/spdystream/tree/v0.2.0) ### Changed -- github.com/Azure/go-autorest/autorest/adal: [v0.8.2 → v0.9.0](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.8.2...v0.9.0) -- github.com/Azure/go-autorest/autorest/date: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.2.0...v0.3.0) -- github.com/Azure/go-autorest/autorest/mocks: [v0.3.0 → v0.4.0](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.3.0...v0.4.0) -- github.com/Azure/go-autorest/autorest: [v0.9.6 → v0.11.1](https://github.com/Azure/go-autorest/autorest/compare/v0.9.6...v0.11.1) -- github.com/Azure/go-autorest/logger: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/logger/compare/v0.1.0...v0.2.0) -- github.com/Azure/go-autorest/tracing: [v0.5.0 → v0.6.0](https://github.com/Azure/go-autorest/tracing/compare/v0.5.0...v0.6.0) -- github.com/Microsoft/hcsshim: [v0.8.9 → 5eafd15](https://github.com/Microsoft/hcsshim/compare/v0.8.9...5eafd15) -- github.com/cilium/ebpf: [9f1617e → 1c8d4c9](https://github.com/cilium/ebpf/compare/9f1617e...1c8d4c9) -- github.com/containerd/cgroups: [bf292b2 → 0dbf7f0](https://github.com/containerd/cgroups/compare/bf292b2...0dbf7f0) -- github.com/coredns/corefile-migration: [v1.0.8 → v1.0.10](https://github.com/coredns/corefile-migration/compare/v1.0.8...v1.0.10) -- github.com/evanphx/json-patch: [e83c0a1 → v4.9.0+incompatible](https://github.com/evanphx/json-patch/compare/e83c0a1...v4.9.0) -- github.com/google/cadvisor: [8450c56 → v0.37.0](https://github.com/google/cadvisor/compare/8450c56...v0.37.0) -- github.com/json-iterator/go: [v1.1.9 → v1.1.10](https://github.com/json-iterator/go/compare/v1.1.9...v1.1.10) -- github.com/opencontainers/go-digest: [v1.0.0-rc1 → v1.0.0](https://github.com/opencontainers/go-digest/compare/v1.0.0-rc1...v1.0.0) -- github.com/opencontainers/runc: [1b94395 → 819fcc6](https://github.com/opencontainers/runc/compare/1b94395...819fcc6) -- github.com/prometheus/client_golang: [v1.6.0 → v1.7.1](https://github.com/prometheus/client_golang/compare/v1.6.0...v1.7.1) -- github.com/prometheus/common: [v0.9.1 → v0.10.0](https://github.com/prometheus/common/compare/v0.9.1...v0.10.0) -- github.com/prometheus/procfs: [v0.0.11 → v0.1.3](https://github.com/prometheus/procfs/compare/v0.0.11...v0.1.3) -- github.com/rubiojr/go-vhd: [0bfd3b3 → 02e2102](https://github.com/rubiojr/go-vhd/compare/0bfd3b3...02e2102) -- github.com/storageos/go-api: [343b3ef → v2.2.0+incompatible](https://github.com/storageos/go-api/compare/343b3ef...v2.2.0) -- github.com/urfave/cli: [v1.22.1 → v1.22.2](https://github.com/urfave/cli/compare/v1.22.1...v1.22.2) -- go.etcd.io/etcd: 54ba958 → dd1b699 -- golang.org/x/crypto: bac4c82 → 75b2880 -- golang.org/x/mod: v0.1.0 → v0.3.0 -- golang.org/x/net: d3edc99 → ab34263 -- golang.org/x/tools: c00d67e → c1934b7 -- k8s.io/kube-openapi: 656914f → 6aeccd4 -- k8s.io/system-validators: v1.1.2 → v1.2.0 -- k8s.io/utils: 6e3d28b → d5654de +- github.com/NYTimes/gziphandler: [56545f4 → v1.1.1](https://github.com/NYTimes/gziphandler/compare/56545f4...v1.1.1) +- github.com/container-storage-interface/spec: [v1.2.0 → v1.3.0](https://github.com/container-storage-interface/spec/compare/v1.2.0...v1.3.0) +- github.com/go-logr/logr: [v0.2.0 → v0.4.0](https://github.com/go-logr/logr/compare/v0.2.0...v0.4.0) +- github.com/gogo/protobuf: [v1.3.1 → v1.3.2](https://github.com/gogo/protobuf/compare/v1.3.1...v1.3.2) +- github.com/kisielk/errcheck: [v1.2.0 → v1.5.0](https://github.com/kisielk/errcheck/compare/v1.2.0...v1.5.0) +- github.com/yuin/goldmark: [v1.1.27 → v1.2.1](https://github.com/yuin/goldmark/compare/v1.1.27...v1.2.1) +- golang.org/x/sync: cd5d95a → 67f06af +- golang.org/x/tools: c1934b7 → 113979e +- k8s.io/klog/v2: v2.4.0 → v2.5.0 +- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15 ### Removed -- github.com/godbus/dbus: [ade71ed](https://github.com/godbus/dbus/tree/ade71ed) -- github.com/xlab/handysort: [fb3537e](https://github.com/xlab/handysort/tree/fb3537e) -- sigs.k8s.io/structured-merge-diff/v3: v3.0.0 -- vbom.ml/util: db5cfe1 +- github.com/docker/spdystream: [449fdfc](https://github.com/docker/spdystream/tree/449fdfc) + + + +# v1.21.0-alpha.2 + + +## Downloads for v1.21.0-alpha.2 + +### Source Code + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes.tar.gz) | 6836f6c8514253fe0831fd171fc4ed92eb6d9a773491c8dc82b90d171a1b10076bd6bfaea56ec1e199c5f46c273265bdb9f174f0b2d99c5af1de4c99b862329e +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-src.tar.gz) | d137694804741a05ab09e5f9a418448b66aba0146c028eafce61bcd9d7c276521e345ce9223ffbc703e8172041d58dfc56a3242a4df3686f24905a4541fcd306 + +### Client binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | 9478b047a97717953f365c13a098feb7e3cb30a3df22e1b82aa945f2208dcc5cb90afc441ba059a3ae7aafb4ee000ec3a52dc65a8c043a5ac7255a391c875330 +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-386.tar.gz) | 44c8dd4b1ddfc256d35786c8abf45b0eb5f0794f5e310d2efc865748adddc50e8bf38aa71295ae8a82884cb65f2e0b9b0737b000f96fd8f2d5c19971d7c4d8e8 +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | e1291989892769de6b978c17b8612b94da6f3b735a4d895100af622ca9ebb968c75548afea7ab00445869625dd0da3afec979e333afbb445805f5d31c1c13cc7 +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | 3c4bcb8cbe73822d68a2f62553a364e20bec56b638c71d0f58679b4f4b277d809142346f18506914e694f6122a3e0f767eab20b7b1c4dbb79e4c5089981ae0f1 +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | 9389974a790268522e187f5ba5237f3ee4684118c7db76bc3d4164de71d8208702747ec333b204c7a78073ab42553cbbce13a1883fab4fec617e093b05fab332 +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | 63399e53a083b5af3816c28ff162c9de6b64c75da4647f0d6bbaf97afdf896823cb1e556f2abac75c6516072293026d3ff9f30676fd75143ac6ca3f4d21f4327 +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | 50898f197a9d923971ff9046c9f02779b57f7b3cea7da02f3ea9bab8c08d65a9c4a7531a2470fa14783460f52111a52b96ebf916c0a1d8215b4070e4e861c1b0 +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-windows-386.tar.gz) | a7743e839e1aa19f5ee20b6ee5000ac8ef9e624ac5be63bb574fad6992e4b9167193ed07e03c9bc524e88bfeed66c95341a38a03bff1b10bc9910345f33019f0 +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | 5f1d19c230bd3542866d16051808d184e9dd3e2f8c001ed4cee7b5df91f872380c2bf56a3add8c9413ead9d8c369efce2bcab4412174df9b823d3592677bf74e + +### Server binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | ef2cac10febde231aeb6f131e589450c560eeaab8046b49504127a091cddc17bc518c2ad56894a6a033033ab6fc6e121b1cc23691683bc36f45fe6b1dd8e0510 +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-arm.tar.gz) | d11c9730307f08e80b2b8a7c64c3e9a9e43c622002e377dfe3a386f4541e24adc79a199a6f280f40298bb36793194fd44ed45defe8a3ee54a9cb1386bc26e905 +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | 28f8c32bf98ee1add7edf5d341c3bac1afc0085f90dcbbfb8b27a92087f13e2b53c327c8935ee29bf1dc3160655b32bbe3e29d5741a8124a3848a777e7d42933 +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | 99ae8d44b0de3518c27fa8bbddd2ecf053dfb789fb9d65f8a4ecf4c8331cf63d2f09a41c2bcd5573247d5f66a1b2e51944379df1715017d920d521b98589508a +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | f8c0e954a2dfc6845614488dadeed069cc7f3f08e33c351d7a77c6ef97867af590932e8576d12998a820a0e4d35d2eee797c764e2810f09ab1e90a5acaeaad33 + +### Node binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | c5456d50bfbe0d75fb150b3662ed7468a0abd3970792c447824f326894382c47bbd3a2cc5a290f691c8c09585ff6fe505ab86b4aff2b7e5ccee11b5e6354ae6c +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-arm.tar.gz) | 335b5cd8672e053302fd94d932fb2fa2e48eeeb1799650b3f93acdfa635e03a8453637569ab710c46885c8317759f4c60aaaf24dca9817d9fa47500fe4a3ca53 +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | 3ee87dbeed8ace9351ac89bdaf7274dd10b4faec3ceba0825f690ec7a2bb7eb7c634274a1065a0939eec8ff3e43f72385f058f4ec141841550109e775bc5eff9 +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | 6956f965b8d719b164214ec9195fdb2c776b907fe6d2c524082f00c27872a73475927fd7d2a994045ce78f6ad2aa5aeaf1eb5514df1810d2cfe342fd4e5ce4a1 +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | 3b643aa905c709c57083c28dd9e8ffd88cb64466cda1499da7fc54176b775003e08b9c7a07b0964064df67c8142f6f1e6c13bfc261bd65fb064049920bfa57d0 +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | b2e6d6fb0091f2541f9925018c2bdbb0138a95bab06b4c6b38abf4b7144b2575422263b78fb3c6fd09e76d90a25a8d35a6d4720dc169794d42c95aa22ecc6d5f + +## Changelog since v1.21.0-alpha.1 + +## Urgent Upgrade Notes + +### (No, really, you MUST read this before you upgrade) + + - Remove storage metrics `storage_operation_errors_total`, since we already have `storage_operation_status_count`.And add new field `status` for `storage_operation_duration_seconds`, so that we can know about all status storage operation latency. ([#98332](https://github.com/kubernetes/kubernetes/pull/98332), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] + +## Changes by Kind + +### Deprecation + +- Remove the TokenRequest and TokenRequestProjection feature gates ([#97148](https://github.com/kubernetes/kubernetes/pull/97148), [@wawa0210](https://github.com/wawa0210)) [SIG Node] +- Removing experimental windows container hyper-v support with Docker ([#97141](https://github.com/kubernetes/kubernetes/pull/97141), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] +- The `export` query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. ([#98312](https://github.com/kubernetes/kubernetes/pull/98312), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth and Testing] + +### API Change + +- Enable SPDY pings to keep connections alive, so that `kubectl exec` and `kubectl port-forward` won't be interrupted. ([#97083](https://github.com/kubernetes/kubernetes/pull/97083), [@knight42](https://github.com/knight42)) [SIG API Machinery and CLI] + +### Documentation + +- Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97935](https://github.com/kubernetes/kubernetes/pull/97935), [@adeniyistephen](https://github.com/adeniyistephen)) [SIG Release and Testing] +- Set kubelet option `--volume-stats-agg-period` to negative value to disable volume calculations. ([#96675](https://github.com/kubernetes/kubernetes/pull/96675), [@pacoxu](https://github.com/pacoxu)) [SIG Node] + +### Bug or Regression + +- Clean ReplicaSet by revision instead of creation timestamp in deployment controller ([#97407](https://github.com/kubernetes/kubernetes/pull/97407), [@waynepeking348](https://github.com/waynepeking348)) [SIG Apps] +- Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. ([#95664](https://github.com/kubernetes/kubernetes/pull/95664), [@DirectXMan12](https://github.com/DirectXMan12)) [SIG API Machinery] +- Fix azure file migration issue ([#97877](https://github.com/kubernetes/kubernetes/pull/97877), [@andyzhangx](https://github.com/andyzhangx)) [SIG Auth, Cloud Provider and Storage] +- Fix kubelet from panic after getting the wrong signal ([#98200](https://github.com/kubernetes/kubernetes/pull/98200), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Fix repeatedly acquire the inhibit lock ([#98088](https://github.com/kubernetes/kubernetes/pull/98088), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Fixed a bug that the kubelet cannot start on BtrfS. ([#98042](https://github.com/kubernetes/kubernetes/pull/98042), [@gjkim42](https://github.com/gjkim42)) [SIG Node] +- Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children ([#98068](https://github.com/kubernetes/kubernetes/pull/98068), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps] +- Fixed no effect namespace when exposing deployment with --dry-run=client. ([#97492](https://github.com/kubernetes/kubernetes/pull/97492), [@masap](https://github.com/masap)) [SIG CLI] +- Fixing a bug where a failed node may not have the NoExecute taint set correctly ([#96876](https://github.com/kubernetes/kubernetes/pull/96876), [@howieyuen](https://github.com/howieyuen)) [SIG Apps and Node] +- Indentation of `Resource Quota` block in kubectl describe namespaces output gets correct. ([#97946](https://github.com/kubernetes/kubernetes/pull/97946), [@dty1er](https://github.com/dty1er)) [SIG CLI] +- KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. ([#98158](https://github.com/kubernetes/kubernetes/pull/98158), [@dougsland](https://github.com/dougsland)) [SIG CLI] +- Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. ([#97372](https://github.com/kubernetes/kubernetes/pull/97372), [@ihgann](https://github.com/ihgann)) [SIG Cluster Lifecycle] +- Kubelet should ignore cgroup driver check on Windows node. ([#97764](https://github.com/kubernetes/kubernetes/pull/97764), [@pacoxu](https://github.com/pacoxu)) [SIG Node and Windows] +- Make podTopologyHints protected by lock ([#95111](https://github.com/kubernetes/kubernetes/pull/95111), [@choury](https://github.com/choury)) [SIG Node] +- Readjust kubelet_containers_per_pod_count bucket ([#98169](https://github.com/kubernetes/kubernetes/pull/98169), [@wawa0210](https://github.com/wawa0210)) [SIG Instrumentation and Node] +- Scores from InterPodAffinity have stronger differentiation. ([#98096](https://github.com/kubernetes/kubernetes/pull/98096), [@leileiwan](https://github.com/leileiwan)) [SIG Scheduling] +- Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. ([#93510](https://github.com/kubernetes/kubernetes/pull/93510), [@smarterclayton](https://github.com/smarterclayton)) [SIG Testing] +- Static pods will be deleted gracefully. ([#98103](https://github.com/kubernetes/kubernetes/pull/98103), [@gjkim42](https://github.com/gjkim42)) [SIG Node] +- Use network.Interface.VirtualMachine.ID to get the binded VM + Skip standalone VM when reconciling LoadBalancer ([#97635](https://github.com/kubernetes/kubernetes/pull/97635), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] + +### Other (Cleanup or Flake) + +- Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' ([#97087](https://github.com/kubernetes/kubernetes/pull/97087), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] +- Migrate generic_scheduler.go and types.go to structured logging. ([#98134](https://github.com/kubernetes/kubernetes/pull/98134), [@tanjing2020](https://github.com/tanjing2020)) [SIG Scheduling] +- Migrate proxy/winuserspace/proxier.go logs to structured logging ([#97941](https://github.com/kubernetes/kubernetes/pull/97941), [@JornShen](https://github.com/JornShen)) [SIG Network] +- Migrate staging/src/k8s.io/apiserver/pkg/audit/policy/reader.go logs to structured logging. ([#98252](https://github.com/kubernetes/kubernetes/pull/98252), [@lala123912](https://github.com/lala123912)) [SIG API Machinery and Auth] +- Migrate staging\src\k8s.io\apiserver\pkg\endpoints logs to structured logging ([#98093](https://github.com/kubernetes/kubernetes/pull/98093), [@lala123912](https://github.com/lala123912)) [SIG API Machinery] +- Node ([#96552](https://github.com/kubernetes/kubernetes/pull/96552), [@pandaamanda](https://github.com/pandaamanda)) [SIG Apps, Cloud Provider, Node and Scheduling] +- The kubectl alpha debug command was scheduled to be removed in v1.21. ([#98111](https://github.com/kubernetes/kubernetes/pull/98111), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] +- Update cri-tools to [v1.20.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.20.0) ([#97967](https://github.com/kubernetes/kubernetes/pull/97967), [@rajibmitra](https://github.com/rajibmitra)) [SIG Cloud Provider] +- Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. ([#98284](https://github.com/kubernetes/kubernetes/pull/98284), [@pjh](https://github.com/pjh)) [SIG Cloud Provider] + +## Dependencies + +### Added +_Nothing has changed._ + +### Changed +- github.com/google/cadvisor: [v0.38.6 → v0.38.7](https://github.com/google/cadvisor/compare/v0.38.6...v0.38.7) +- k8s.io/gengo: 83324d8 → b6c5ce2 + +### Removed +_Nothing has changed._ + + + +# v1.21.0-alpha.1 + + +## Downloads for v1.21.0-alpha.1 + +### Source Code + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes.tar.gz) | b2bacd5c3fc9f829e6269b7d2006b0c6e464ff848bb0a2a8f2fe52ad2d7c4438f099bd8be847d8d49ac6e4087f4d74d5c3a967acd798e0b0cb4d7a2bdb122997 +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-src.tar.gz) | 518ac5acbcf23902fb1b902b69dbf3e86deca5d8a9b5f57488a15f185176d5a109558f3e4df062366af874eca1bcd61751ee8098b0beb9bcdc025d9a1c9be693 + +### Client binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | eaa7aea84a5ed954df5ec710cbeb6ec88b46465f43cb3d09aabe2f714b84a050a50bf5736089f09dbf1090f2e19b44823d656c917e3c8c877630756c3026f2b6 +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-386.tar.gz) | 47f74b8d46ad1779c5b0b5f15aa15d5513a504eeb6f53db4201fbe9ff8956cb986b7c1b0e9d50a99f78e9e2a7f304f3fc1cc2fa239296d9a0dd408eb6069e975 +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | 1a148e282628b008c8abd03dd12ec177ced17584b5115d92cd33dd251e607097d42e9da8c7089bd947134b900f85eb75a4740b6a5dd580c105455b843559df39 +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | d13d2feb73bd032dc01f7e2955b98d8215a39fe1107d037a73fa1f7d06c3b93ebaa53ed4952d845c64454ef3cca533edb97132d234d50b6fb3bcbd8a8ad990eb +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | 8252105a17b09a78e9ad2c024e4e401a69764ac869708a071aaa06f81714c17b9e7c5b2eb8efde33f24d0b59f75c5da607d5e1e72bdf12adfbb8c829205cd1c1 +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | 297a9082df4988389dc4be30eb636dff49f36f5d87047bab44745884e610f46a17ae3a08401e2cab155b7c439f38057bfd8288418215f7dd3bf6a49dbe61ea0e +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | 04c06490dd17cd5dccfd92bafa14acf64280ceaea370d9635f23aeb6984d1beae6d0d1d1506edc6f30f927deeb149b989d3e482b47fbe74008b371f629656e79 +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-windows-386.tar.gz) | ec6e9e87a7d685f8751d7e58f24f417753cff5554a7229218cb3a08195d461b2e12409344950228e9fbbc92a8a06d35dd86242da6ff1e6652ec1fae0365a88c1 +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | 51039e6221d3126b5d15e797002ae01d4f0b10789c5d2056532f27ef13f35c5a2e51be27764fda68e8303219963126559023aed9421313bec275c0827fbcaf8a + +### Server binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | 4edf820930c88716263560275e3bd7fadb8dc3700b9f8e1d266562e356e0abeb1a913f536377dab91218e3940b447d6bf1da343b85da25c2256dc4dcde5798dd +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-arm.tar.gz) | b15213e53a8ab4ba512ce6ef9ad42dd197d419c61615cd23de344227fd846c90448d8f3d98e555b63ba5b565afa627cca6b7e3990ebbbba359c96f2391302df1 +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | 5be29cca9a9358fc68351ee63e99d57dc2ffce6e42fc3345753dbbf7542ff2d770c4852424158540435fa6e097ce3afa9b13affc40c8b3b69fe8406798f8068f +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | 89fd99ab9ce85db0b94b86709932105efc883cc93959cf7ea9a39e79a4acea23064d7010eeb577450cccabe521c04b7ba47bbec212ed37edeed7cb04bad34518 +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | 2fbc30862c77d247aa8d96ab9d1a144599505287b0033a3a2d0988958e7bb2f2e8b67f52c1fec74b4ec47d74ba22cd0f6cb5c4228acbaa72b1678d5fece0254d + +### Node binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | 95658d321a0a371c0900b401d1469d96915310afbc4e4b9b11f031438bb188513b57d5a60b5316c3b0c18f541cda6f0ac42f59a76495f8abc743a067115da23a +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-arm.tar.gz) | f375acfb42aad6c65b833c270e7e3acfe9cd1d6b2441c33874e77faae263957f7acfe86f1b71f14298118595e4cc6952c7dea0c832f7f2e72428336f13034362 +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | 43b4baccd58d74e7f48d096ab92f2bbbcdf47e30e7a3d2b56c6cc9f90002cfd4fefaac894f69bd5f9f4dbdb09a4749a77eb76b1b97d91746bd96fe94457879ab +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | e7962b522c6c7c14b9ee4c1d254d8bdd9846b2b33b0443fc9c4a41be6c40e5e6981798b720f0148f36263d5cc45d5a2bb1dd2f9ab2838e3d002e45b9bddeb7bf +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | 49ebc97f01829e65f7de15be00b882513c44782eaadd1b1825a227e3bd3c73cc6aca8345af05b303d8c43aa2cb944a069755b2709effb8cc22eae621d25d4ba5 +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.21.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | 6e0fd7724b09e6befbcb53b33574e97f2db089f2eee4bbf391abb7f043103a5e6e32e3014c0531b88f9a3ca88887bbc68625752c44326f98dd53adb3a6d1bed8 + +## Changelog since v1.20.0 + +## Urgent Upgrade Notes + +### (No, really, you MUST read this before you upgrade) + + - Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. ([#92938](https://github.com/kubernetes/kubernetes/pull/92938), [@lbernail](https://github.com/lbernail)) [SIG Network and Release] + +## Changes by Kind + +### Deprecation + +- Deprecate the `topologyKeys` field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. ([#96736](https://github.com/kubernetes/kubernetes/pull/96736), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] +- Kubeadm: deprecated command "alpha selfhosting pivot" is removed now. ([#97627](https://github.com/kubernetes/kubernetes/pull/97627), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] +- Kubeadm: graduate the command `kubeadm alpha kubeconfig user` to `kubeadm kubeconfig user`. The `kubeadm alpha kubeconfig user` command is deprecated now. ([#97583](https://github.com/kubernetes/kubernetes/pull/97583), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] +- Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. ([#97706](https://github.com/kubernetes/kubernetes/pull/97706), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] +- Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. ([#96447](https://github.com/kubernetes/kubernetes/pull/96447), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing] +- The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. ([#97171](https://github.com/kubernetes/kubernetes/pull/97171), [@deads2k](https://github.com/deads2k)) [SIG Auth and CLI] + +### API Change + +- Change the APIVersion proto name of BoundObjectRef from aPIVersion to apiVersion. ([#97379](https://github.com/kubernetes/kubernetes/pull/97379), [@kebe7jun](https://github.com/kebe7jun)) [SIG Auth] +- Promote Immutable Secrets/ConfigMaps feature to Stable. + This allows to set `Immutable` field in Secrets or ConfigMap object to mark their contents as immutable. ([#97615](https://github.com/kubernetes/kubernetes/pull/97615), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps, Architecture, Node and Testing] + +### Feature + +- Add flag --lease-max-object-size and metric etcd_lease_object_counts for kube-apiserver to config and observe max objects attached to a single etcd lease. ([#97480](https://github.com/kubernetes/kubernetes/pull/97480), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery, Instrumentation and Scalability] +- Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. ([#97009](https://github.com/kubernetes/kubernetes/pull/97009), [@lingsamuel](https://github.com/lingsamuel)) [SIG API Machinery and Scalability] +- Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. ([#96502](https://github.com/kubernetes/kubernetes/pull/96502), [@249043822](https://github.com/249043822)) [SIG Auth] +- Kubeadm now includes CoreDNS v1.8.0. ([#96429](https://github.com/kubernetes/kubernetes/pull/96429), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] +- Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. ([#97266](https://github.com/kubernetes/kubernetes/pull/97266), [@robbiemcmichael](https://github.com/robbiemcmichael)) [SIG Cluster Lifecycle] +- Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional ([#96378](https://github.com/kubernetes/kubernetes/pull/96378), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle and Node] +- The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. ([#97322](https://github.com/kubernetes/kubernetes/pull/97322), [@claudiubelu](https://github.com/claudiubelu)) [SIG Windows] +- The apimachinery util/net function used to detect the bind address `ResolveBindAddress()` + takes into consideration global ip addresses on loopback interfaces when: + - the host has default routes + - there are no global IPs on those interfaces. + in order to support more complex network scenarios like BGP Unnumbered RFC 5549 ([#95790](https://github.com/kubernetes/kubernetes/pull/95790), [@aojea](https://github.com/aojea)) [SIG Network] + +### Bug or Regression + +- ## Changelog + + ### General + - Fix priority expander falling back to a random choice even though there is a higher priority option to choose + - Clone `kubernetes/kubernetes` in `update-vendor.sh` shallowly, instead of fetching all revisions + - Speed up binpacking by reducing the number of PreFilter calls (call once per pod instead of #pods*#nodes times) + - Speed up finding unneeded nodes by 5x+ in very large clusters by reducing the number of PreFilter calls + - Expose `--max-nodes-total` as a metric + - Errors in `IncreaseSize` changed from type `apiError` to `cloudProviderError` + - Make `build-in-docker` and `test-in-docker` work on Linux systems with SELinux enabled + - Fix an error where existing nodes were not considered as destinations while finding place for pods in scale-down simulations + - Remove redundant log lines and reduce severity around parsing kubeEnv + - Don't treat nodes created by virtual kubelet as nodes from non-autoscaled node groups + - Remove redundant logging around calculating node utilization + - Add configurable `--network` and `--rm` flags for docker in `Makefile` + - Subtract DaemonSet pods' requests from node allocatable in the denominator while computing node utilization + - Include taints by condition when determining if a node is unready/still starting + - Fix `update-vendor.sh` to work on OSX and zsh + - Add best-effort eviction for DaemonSet pods while scaling down non-empty nodes + - Add build support for ARM64 + + ### AliCloud + - Add missing daemonsets and replicasets to ALI example cluster role + + ### Apache CloudStack + - Add support for Apache CloudStack + + ### AWS + - Regenerate list of EC2 instances + - Fix pricing endpoint in AWS China Region + + ### Azure + - Add optional jitter on initial VMSS VM cache refresh, keep the refreshes spread over time + - Serve from cache for the whole period of ongoing throttling + - Fix unwanted VMSS VMs cache invalidations + - Enforce setting the number of retries if cloud provider backoff is enabled + - Don't update capacity if VMSS provisioning state is updating + - Support allocatable resources overrides via VMSS tags + - Add missing stable labels in template nodes + - Proactively set instance status to deleting on node deletions + + ### Cluster API + - Migrate interaction with the API from using internal types to using Unstructured + - Improve tests to work better with constrained resources + - Add support for node autodiscovery + - Add support for `--cloud-config` + - Update group identifier to use for Cluster API annotations + + ### Exoscale + - Add support for Exoscale + + ### GCE + - Decrease the number of GCE Read Requests made while deleting nodes + - Base pricing of custom instances on their instance family type + - Add pricing information for missing machine types + - Add pricing information for different GPU types + - Ignore the new `topology.gke.io/zone` label when comparing groups + - Add missing stable labels to template nodes + + ### HuaweiCloud + - Add auto scaling group support + - Implement node group by AS + - Implement getting desired instance number of node group + - Implement increasing node group size + - Implement TemplateNodeInfo + - Implement caching instances + + ### IONOS + - Add support for IONOS + + ### Kubemark + - Skip non-kubemark nodes while computing node infos for node groups. + + ### Magnum + - Add Magnum support in the Cluster Autoscaler helm chart + + ### Packet + - Allow empty nodepools + - Add support for multiple nodepools + - Add pricing support + + ## Image + Image: `k8s.gcr.io/autoscaling/cluster-autoscaler:v1.20.0` ([#97011](https://github.com/kubernetes/kubernetes/pull/97011), [@towca](https://github.com/towca)) [SIG Cloud Provider] +- AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. ([#96873](https://github.com/kubernetes/kubernetes/pull/96873), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node] +- Add limited lines to log when having tail option ([#93920](https://github.com/kubernetes/kubernetes/pull/93920), [@zhouya0](https://github.com/zhouya0)) [SIG Node] +- Avoid systemd-logind loading configuration warning ([#97950](https://github.com/kubernetes/kubernetes/pull/97950), [@wzshiming](https://github.com/wzshiming)) [SIG Node] +- Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs ([#97029](https://github.com/kubernetes/kubernetes/pull/97029), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider and Testing] +- Copy annotations with empty value when deployment rolls back ([#94858](https://github.com/kubernetes/kubernetes/pull/94858), [@waynepeking348](https://github.com/waynepeking348)) [SIG Apps] +- Detach volumes from vSphere nodes not tracked by attach-detach controller ([#96689](https://github.com/kubernetes/kubernetes/pull/96689), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] +- Fix kubectl label error when local=true is set. ([#97440](https://github.com/kubernetes/kubernetes/pull/97440), [@pandaamanda](https://github.com/pandaamanda)) [SIG CLI] +- Fix Azure file share not deleted issue when the namespace is deleted ([#97417](https://github.com/kubernetes/kubernetes/pull/97417), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] +- Fix CVE-2020-8555 for Gluster client connections. ([#97922](https://github.com/kubernetes/kubernetes/pull/97922), [@liggitt](https://github.com/liggitt)) [SIG Storage] +- Fix counting error in service/nodeport/loadbalancer quota check ([#97451](https://github.com/kubernetes/kubernetes/pull/97451), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Network and Testing] +- Fix kubectl-convert import known versions ([#97754](https://github.com/kubernetes/kubernetes/pull/97754), [@wzshiming](https://github.com/wzshiming)) [SIG CLI and Testing] +- Fix missing cadvisor machine metrics. ([#97006](https://github.com/kubernetes/kubernetes/pull/97006), [@lingsamuel](https://github.com/lingsamuel)) [SIG Node] +- Fix nil VMSS name when setting service to auto mode ([#97366](https://github.com/kubernetes/kubernetes/pull/97366), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable ([#95269](https://github.com/kubernetes/kubernetes/pull/95269), [@SataQiu](https://github.com/SataQiu)) [SIG Node] +- Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. ([#97980](https://github.com/kubernetes/kubernetes/pull/97980), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node] +- Fix to recover CSI volumes from certain dangling attachments ([#96617](https://github.com/kubernetes/kubernetes/pull/96617), [@yuga711](https://github.com/yuga711)) [SIG Apps and Storage] +- Fix: azure file latency issue for metadata-heavy workloads ([#97082](https://github.com/kubernetes/kubernetes/pull/97082), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] +- Fixed Cinder volume IDs on OpenStack Train ([#96673](https://github.com/kubernetes/kubernetes/pull/96673), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider] +- Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. ([#97013](https://github.com/kubernetes/kubernetes/pull/97013), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] +- Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. ([#97174](https://github.com/kubernetes/kubernetes/pull/97174), [@hanlins](https://github.com/hanlins)) [SIG Node] +- Fixed bug in CPUManager with race on container map access ([#97427](https://github.com/kubernetes/kubernetes/pull/97427), [@klueska](https://github.com/klueska)) [SIG Node] +- Fixed cleanup of block devices when /var/lib/kubelet is a symlink. ([#96889](https://github.com/kubernetes/kubernetes/pull/96889), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] +- GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. ([#97740](https://github.com/kubernetes/kubernetes/pull/97740), [@prameshj](https://github.com/prameshj)) [SIG Cloud Provider and Network] +- Ignore update pod with no new images in alwaysPullImages admission controller ([#96668](https://github.com/kubernetes/kubernetes/pull/96668), [@pacoxu](https://github.com/pacoxu)) [SIG Apps, Auth and Node] +- Kubeadm now installs version 3.4.13 of etcd when creating a cluster with v1.19 ([#97244](https://github.com/kubernetes/kubernetes/pull/97244), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] +- Kubeadm: avoid detection of the container runtime for commands that do not need it ([#97625](https://github.com/kubernetes/kubernetes/pull/97625), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms ([#97403](https://github.com/kubernetes/kubernetes/pull/97403), [@abelbarrera15](https://github.com/abelbarrera15)) [SIG Cluster Lifecycle] +- Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. ([#97919](https://github.com/kubernetes/kubernetes/pull/97919), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Performance regression [#97685](https://github.com/kubernetes/kubernetes/issues/97685) has been fixed. ([#97860](https://github.com/kubernetes/kubernetes/pull/97860), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] +- Remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS ([#97336](https://github.com/kubernetes/kubernetes/pull/97336), [@maaoBit](https://github.com/maaoBit)) [SIG Network] +- The current version of the container image publicly exposed IP serving a /metrics endpoint to the Internet. The new version of the container image serves /metrics endpoint on a different port. ([#97621](https://github.com/kubernetes/kubernetes/pull/97621), [@vbannai](https://github.com/vbannai)) [SIG Cloud Provider] +- Use force unmount for NFS volumes if regular mount fails after 1 minute timeout ([#96844](https://github.com/kubernetes/kubernetes/pull/96844), [@gnufied](https://github.com/gnufied)) [SIG Storage] +- Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. ([#92817](https://github.com/kubernetes/kubernetes/pull/92817), [@kmala](https://github.com/kmala)) [SIG Node] +- Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. ([#97857](https://github.com/kubernetes/kubernetes/pull/97857), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Auth] +- Warning about using a deprecated volume plugin is logged only once. ([#96751](https://github.com/kubernetes/kubernetes/pull/96751), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] + +### Other (Cleanup or Flake) + +- Bump github.com/Azure/go-autorest/autorest to v0.11.12 ([#97033](https://github.com/kubernetes/kubernetes/pull/97033), [@patrickshan](https://github.com/patrickshan)) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle] +- Delete deprecated mixed protocol annotation ([#97096](https://github.com/kubernetes/kubernetes/pull/97096), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] +- Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. ([#96296](https://github.com/kubernetes/kubernetes/pull/96296), [@aojea](https://github.com/aojea)) [SIG Network and Testing] +- Kubeadm: fix a whitespace issue in the output of the "kubeadm join" command shown as the output of "kubeadm init" and "kubeadm token create --print-join-command" ([#97413](https://github.com/kubernetes/kubernetes/pull/97413), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] +- Kubeadm: improve the error messaging when the user provides an invalid discovery token CA certificate hash. ([#97290](https://github.com/kubernetes/kubernetes/pull/97290), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] +- Migrate log messages in pkg/scheduler/{scheduler.go,factory.go} to structured logging ([#97509](https://github.com/kubernetes/kubernetes/pull/97509), [@aldudko](https://github.com/aldudko)) [SIG Scheduling] +- Migrate proxy/iptables/proxier.go logs to structured logging ([#97678](https://github.com/kubernetes/kubernetes/pull/97678), [@JornShen](https://github.com/JornShen)) [SIG Network] +- Migrate some scheduler log messages to structured logging ([#97349](https://github.com/kubernetes/kubernetes/pull/97349), [@aldudko](https://github.com/aldudko)) [SIG Scheduling] +- NONE ([#97167](https://github.com/kubernetes/kubernetes/pull/97167), [@geegeea](https://github.com/geegeea)) [SIG Node] +- NetworkPolicy validation framework optimizations for rapidly verifying CNI's work correctly across several pods and namespaces ([#91592](https://github.com/kubernetes/kubernetes/pull/91592), [@jayunit100](https://github.com/jayunit100)) [SIG Network, Storage and Testing] +- Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. ([#97618](https://github.com/kubernetes/kubernetes/pull/97618), [@jherrera123](https://github.com/jherrera123)) [SIG Release and Testing] +- Scheduler plugin validation now provides all errors detected instead of the first one. ([#96745](https://github.com/kubernetes/kubernetes/pull/96745), [@lingsamuel](https://github.com/lingsamuel)) [SIG Node, Scheduling and Testing] +- Storage related e2e testsuite redesign & cleanup ([#96573](https://github.com/kubernetes/kubernetes/pull/96573), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Storage and Testing] +- The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. ([#97693](https://github.com/kubernetes/kubernetes/pull/97693), [@enj](https://github.com/enj)) [SIG API Machinery and Auth] +- The `AttachVolumeLimit` feature gate that is GA since v1.17 is now removed. ([#96539](https://github.com/kubernetes/kubernetes/pull/96539), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Storage] +- The `CSINodeInfo` feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the `--feature-gates` argument. ([#96561](https://github.com/kubernetes/kubernetes/pull/96561), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Apps, Auth, Scheduling, Storage and Testing] +- The deprecated feature gates `RotateKubeletClientCertificate`, `AttachVolumeLimit`, `VolumePVCDataSource` and `EvenPodsSpread` are now unconditionally enabled and can no longer be specified in component invocations. ([#97306](https://github.com/kubernetes/kubernetes/pull/97306), [@gavinfish](https://github.com/gavinfish)) [SIG Node, Scheduling and Storage] +- `ServiceNodeExclusion`, `NodeDisruptionExclusion` and `LegacyNodeRoleBehavior`(locked to false) features have been promoted to GA. + To prevent control plane nodes being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. ([#97543](https://github.com/kubernetes/kubernetes/pull/97543), [@pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Apps, Cloud Provider and Network] + +### Uncategorized + +- Adding Brazilian Portuguese translation for kubectl ([#61595](https://github.com/kubernetes/kubernetes/pull/61595), [@cpanato](https://github.com/cpanato)) [SIG CLI] + +## Dependencies + +### Added +_Nothing has changed._ + +### Changed +- github.com/Azure/go-autorest/autorest: [v0.11.1 → v0.11.12](https://github.com/Azure/go-autorest/autorest/compare/v0.11.1...v0.11.12) +- github.com/coredns/corefile-migration: [v1.0.10 → v1.0.11](https://github.com/coredns/corefile-migration/compare/v1.0.10...v1.0.11) +- github.com/golang/mock: [v1.4.1 → v1.4.4](https://github.com/golang/mock/compare/v1.4.1...v1.4.4) +- github.com/google/cadvisor: [v0.38.5 → v0.38.6](https://github.com/google/cadvisor/compare/v0.38.5...v0.38.6) +- github.com/heketi/heketi: [c2e2a4a → v10.2.0+incompatible](https://github.com/heketi/heketi/compare/c2e2a4a...v10.2.0) +- github.com/miekg/dns: [v1.1.4 → v1.1.35](https://github.com/miekg/dns/compare/v1.1.4...v1.1.35) +- k8s.io/system-validators: v1.2.0 → v1.3.0 + +### Removed +- rsc.io/quote/v3: v3.1.0 +- rsc.io/sampler: v1.3.0 From e955fd5d0f7f4983d83e511a22be139bb90da2b3 Mon Sep 17 00:00:00 2001 From: Rey Lejano Date: Thu, 8 Apr 2021 13:30:49 -0700 Subject: [PATCH 060/218] update config.toml --- config.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config.toml b/config.toml index f4922ef075..170e6b1091 100644 --- a/config.toml +++ b/config.toml @@ -189,7 +189,7 @@ fullversion = "v1.20.5" version = "v1.20" githubbranch = "v1.20.5" docsbranch = "release-1.20" -url = "https://v1-20.kubernetes.io" +url = "https://v1-20.docs.kubernetes.io" [[params.versions]] fullversion = "v1.19.9" From ff6aa6e3b4751df431ec7121f4f009e66e04cec4 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sat, 3 Apr 2021 16:43:49 +0100 Subject: [PATCH 061/218] Mention API extensibility as a front-page feature Kubernetes has really good options for extending and customizing its behavior. Mention these right on the front page. --- content/en/docs/concepts/extend-kubernetes/_index.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/en/docs/concepts/extend-kubernetes/_index.md b/content/en/docs/concepts/extend-kubernetes/_index.md index cc5ba809ec..a2d24e3cb2 100644 --- a/content/en/docs/concepts/extend-kubernetes/_index.md +++ b/content/en/docs/concepts/extend-kubernetes/_index.md @@ -7,6 +7,10 @@ reviewers: - lavalamp - cheftako - chenopis +feature: + title: Designed for extensibility + description: > + Add features to your Kubernetes cluster without changing upstream source code. content_type: concept no_list: true --- From d2c9a401396e6732157e54d1391747bcb36d62a2 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 6 Apr 2021 19:19:16 +0100 Subject: [PATCH 062/218] Document Pod name label for StatefulSet Add a missing well-known label to our list. --- .../en/docs/reference/labels-annotations-taints.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/content/en/docs/reference/labels-annotations-taints.md b/content/en/docs/reference/labels-annotations-taints.md index 08861e5811..d2dc4359aa 100644 --- a/content/en/docs/reference/labels-annotations-taints.md +++ b/content/en/docs/reference/labels-annotations-taints.md @@ -99,6 +99,18 @@ See [topology.kubernetes.io/zone](#topologykubernetesiozone). {{< note >}} Starting in v1.17, this label is deprecated in favor of [topology.kubernetes.io/zone](#topologykubernetesiozone). {{< /note >}} +## statefulset.kubernetes.io/pod-name {#statefulsetkubernetesiopod-name} + +Example: + +`statefulset.kubernetes.io/pod-name=mystatefulset-7` + +When a StatefulSet controller creates a Pod for the StatefulSet, the control plane +sets this label on that Pod. The value of the label is the name of the Pod being created. + +See [Pod Name Label](/docs/concepts/workloads/controllers/statefulset/#pod-name-label) in the +StatefulSet topic for more details. + ## topology.kubernetes.io/region {#topologykubernetesioregion} Example: From 6214d894455a5cc7af77ada09b976970fa063df7 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sun, 28 Mar 2021 20:04:57 +0100 Subject: [PATCH 063/218] Run local container previews with minimal privileges --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 12b6f67d5a..b905a1c337 100644 --- a/Makefile +++ b/Makefile @@ -68,7 +68,7 @@ container-build: module-check $(CONTAINER_RUN) --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 $(CONTAINER_IMAGE) sh -c "npm ci && hugo --minify" container-serve: module-check ## Boot the development server using container. Run `make container-image` before this. - $(CONTAINER_RUN) --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 -p 1313:1313 $(CONTAINER_IMAGE) hugo server --buildFuture --bind 0.0.0.0 --destination /tmp/hugo --cleanDestinationDir + $(CONTAINER_RUN) --cap-drop=ALL --cap-add=AUDIT_WRITE --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 -p 1313:1313 $(CONTAINER_IMAGE) hugo server --buildFuture --bind 0.0.0.0 --destination /tmp/hugo --cleanDestinationDir test-examples: scripts/test_examples.sh install From d6589e0ce3d4265ac311f6e381d7bb4eeb70a0d2 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sat, 3 Apr 2021 17:05:03 +0100 Subject: [PATCH 064/218] =?UTF-8?q?Omit=20=E2=80=9CEdit=20this=20page?= =?UTF-8?q?=E2=80=9D=20links=20for=20generated=20content?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copy the page meta links template from the Docsy theme, and adapt it to only suggest editing pages that aren't autogenerated. --- layouts/partials/page-meta-links.html | 41 +++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 layouts/partials/page-meta-links.html diff --git a/layouts/partials/page-meta-links.html b/layouts/partials/page-meta-links.html new file mode 100644 index 0000000000..97e84b55ae --- /dev/null +++ b/layouts/partials/page-meta-links.html @@ -0,0 +1,41 @@ +{{/* template adapted from Docsy theme */}} +{{ if .Path }} + {{ $pathFormatted := replace .Path "\\" "/" }} + {{ $gh_repo := ($.Param "github_repo") }} + {{ $gh_subdir := ($.Param "github_subdir") }} + {{ $gh_project_repo := ($.Param "github_project_repo") }} + {{ $gh_branch := (default "master" ($.Param "github_branch")) }} +
+ {{ if $gh_repo }} + {{ $gh_repo_path := printf "%s/content/%s" $gh_branch $pathFormatted }} + {{ if and ($gh_subdir) (.Site.Language.Lang) }} + {{ $gh_repo_path = printf "%s/%s/content/%s/%s" $gh_branch $gh_subdir ($.Site.Language.Lang) $pathFormatted }} + {{ else if .Site.Language.Lang }} + {{ $gh_repo_path = printf "%s/content/%s/%s" $gh_branch ($.Site.Language.Lang) $pathFormatted }} + {{ else if $gh_subdir }} + {{ $gh_repo_path = printf "%s/%s/content/%s" $gh_branch $gh_subdir $pathFormatted }} + {{ end }} + {{ $editURL := printf "%s/edit/%s" $gh_repo $gh_repo_path }} + {{ $createURL := printf "%s/edit/%s" $gh_repo $gh_repo_path }} + {{ $issuesURL := printf "%s/issues/new?title=%s" $gh_repo (htmlEscape $.Title )}} + {{ $newPageStub := resources.Get "stubs/new-page-template.md" }} + {{ $newPageQS := querify "value" $newPageStub.Content "filename" "change-me.md" | safeURL }} + {{ $newPageURL := printf "%s/new/%s?%s" $gh_repo $gh_repo_path $newPageQS }} + + {{ if not (.Param "auto_generated") }} + {{ T "post_edit_this" }} + {{ T "post_create_child_page" }} + {{ end }} + + {{ T "post_create_issue" }} + {{ if $gh_project_repo }} + {{ $project_issueURL := printf "%s/issues/new" $gh_project_repo }} + {{ T "post_create_project_issue" }} + {{ end }} + + {{ end }} + {{ with .CurrentSection.AlternativeOutputFormats.Get "print" }} + {{ T "print_entire_section" }} + {{ end }} +
+{{ end }} From c5bab50cafa5a0694a2eb16eb59eb0f11188b406 Mon Sep 17 00:00:00 2001 From: Han Kang Date: Mon, 5 Apr 2021 12:42:28 -0700 Subject: [PATCH 065/218] add back the link to stable metrics list --- .../en/docs/concepts/cluster-administration/system-metrics.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/en/docs/concepts/cluster-administration/system-metrics.md b/content/en/docs/concepts/cluster-administration/system-metrics.md index 9852f9cc99..bfcfec9ecb 100644 --- a/content/en/docs/concepts/cluster-administration/system-metrics.md +++ b/content/en/docs/concepts/cluster-administration/system-metrics.md @@ -174,4 +174,5 @@ Here is an example: ## {{% heading "whatsnext" %}} * Read about the [Prometheus text format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) for metrics +* See the list of [stable Kubernetes metrics](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml) * Read about the [Kubernetes deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior) From 937f848340bf5362ddbf183933a0d4dbf1d4574a Mon Sep 17 00:00:00 2001 From: Qiming Teng Date: Fri, 9 Apr 2021 08:47:00 +0800 Subject: [PATCH 066/218] API reference for v1.21 --- .../generated/kubernetes-api/v1.21/index.html | 1129 +++++++++++------ .../kubernetes-api/v1.21/js/navData.js | 2 +- 2 files changed, 774 insertions(+), 357 deletions(-) diff --git a/static/docs/reference/generated/kubernetes-api/v1.21/index.html b/static/docs/reference/generated/kubernetes-api/v1.21/index.html index ddab130403..8eee575dd4 100644 --- a/static/docs/reference/generated/kubernetes-api/v1.21/index.html +++ b/static/docs/reference/generated/kubernetes-api/v1.21/index.html @@ -25,35 +25,35 @@
  • Container v1 core