From b4a56597b03084183bc547a5e349f3bf011839f7 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 18 Jan 2022 20:33:43 +0200 Subject: [PATCH] kubeadm: document known label, taints, annotations Include information about the kubeadm managed labels, taints and annotation in the main page on this topic. Add two sections for core k8s and kubeadm. Adjust markdown headings to have "##" as the top level for these sections. --- .../reference/labels-annotations-taints.md | 182 ++++++++++++------ 1 file changed, 123 insertions(+), 59 deletions(-) diff --git a/content/en/docs/reference/labels-annotations-taints.md b/content/en/docs/reference/labels-annotations-taints.md index 453e50fee0..cb99bdf592 100644 --- a/content/en/docs/reference/labels-annotations-taints.md +++ b/content/en/docs/reference/labels-annotations-taints.md @@ -10,11 +10,11 @@ Kubernetes reserves all labels and annotations in the kubernetes.io namespace. This document serves both as a reference to the values and as a coordination point for assigning values. - - -## kubernetes.io/arch +## Core + +### kubernetes.io/arch Example: `kubernetes.io/arch=amd64` @@ -22,7 +22,7 @@ Used on: Node The Kubelet populates this with `runtime.GOARCH` as defined by Go. This can be handy if you are mixing arm and x86 nodes. -## kubernetes.io/os +### kubernetes.io/os Example: `kubernetes.io/os=linux` @@ -30,28 +30,28 @@ Used on: Node The Kubelet populates this with `runtime.GOOS` as defined by Go. This can be handy if you are mixing operating systems in your cluster (for example: mixing Linux and Windows nodes). -## kubernetes.io/metadata.name +### kubernetes.io/metadata.name Example: `kubernetes.io/metadata.name=mynamespace` Used on: Namespaces -The Kubernetes API server (part of the {{< glossary_tooltip text="control plane" term_id="control-plane" >}}) +The Kubernetes API server (part of the {{< glossary_tooltip text="control plane" term_id="control-plane" >}}) sets this label on all namespaces. The label value is set -to the name of the namespace. You can't change this label's value. +to the name of the namespace. You can't change this label's value. This is useful if you want to target a specific namespace with a label {{< glossary_tooltip text="selector" term_id="selector" >}}. -## beta.kubernetes.io/arch (deprecated) +### beta.kubernetes.io/arch (deprecated) This label has been deprecated. Please use `kubernetes.io/arch` instead. -## beta.kubernetes.io/os (deprecated) +### beta.kubernetes.io/os (deprecated) This label has been deprecated. Please use `kubernetes.io/os` instead. -## kubernetes.io/hostname {#kubernetesiohostname} +### kubernetes.io/hostname {#kubernetesiohostname} Example: `kubernetes.io/hostname=ip-172-20-114-199.ec2.internal` @@ -62,17 +62,17 @@ The Kubelet populates this label with the hostname. Note that the hostname can b This label is also used as part of the topology hierarchy. See [topology.kubernetes.io/zone](#topologykubernetesiozone) for more information. -## kubernetes.io/change-cause {#change-cause} +### kubernetes.io/change-cause {#change-cause} Example: `kubernetes.io/change-cause=kubectl edit --record deployment foo` Used on: All Objects -This annotation is a best guess at why something was changed. +This annotation is a best guess at why something was changed. It is populated when adding `--record` to a `kubectl` command that may change an object. -## kubernetes.io/description {#description} +### kubernetes.io/description {#description} Example: `kubernetes.io/description: "Description of K8s object."` @@ -80,7 +80,7 @@ Used on: All Objects This annotation is used for describing specific behaviour of given object. -## controller.kubernetes.io/pod-deletion-cost {#pod-deletion-cost} +### controller.kubernetes.io/pod-deletion-cost {#pod-deletion-cost} Example: `controller.kubernetes.io/pod-deletion-cost=10` @@ -89,11 +89,11 @@ Used on: Pod This annotation is used to set [Pod Deletion Cost](/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost) which allows users to influence ReplicaSet downscaling order. The annotation parses into an `int32` type. -## beta.kubernetes.io/instance-type (deprecated) +### beta.kubernetes.io/instance-type (deprecated) {{< note >}} Starting in v1.17, this label is deprecated in favor of [node.kubernetes.io/instance-type](#nodekubernetesioinstance-type). {{< /note >}} -## node.kubernetes.io/instance-type {#nodekubernetesioinstance-type} +### node.kubernetes.io/instance-type {#nodekubernetesioinstance-type} Example: `node.kubernetes.io/instance-type=m3.medium` @@ -104,19 +104,19 @@ This will be set only if you are using a `cloudprovider`. This setting is handy if you want to target certain workloads to certain instance types, but typically you want to rely on the Kubernetes scheduler to perform resource-based scheduling. You should aim to schedule based on properties rather than on instance types (for example: require a GPU, instead of requiring a `g2.2xlarge`). -## failure-domain.beta.kubernetes.io/region (deprecated) {#failure-domainbetakubernetesioregion} +### failure-domain.beta.kubernetes.io/region (deprecated) {#failure-domainbetakubernetesioregion} See [topology.kubernetes.io/region](#topologykubernetesioregion). {{< note >}} Starting in v1.17, this label is deprecated in favor of [topology.kubernetes.io/region](#topologykubernetesioregion). {{< /note >}} -## failure-domain.beta.kubernetes.io/zone (deprecated) {#failure-domainbetakubernetesiozone} +### failure-domain.beta.kubernetes.io/zone (deprecated) {#failure-domainbetakubernetesiozone} See [topology.kubernetes.io/zone](#topologykubernetesiozone). {{< note >}} Starting in v1.17, this label is deprecated in favor of [topology.kubernetes.io/zone](#topologykubernetesiozone). {{< /note >}} -## statefulset.kubernetes.io/pod-name {#statefulsetkubernetesiopod-name} +### statefulset.kubernetes.io/pod-name {#statefulsetkubernetesiopod-name} Example: @@ -128,7 +128,7 @@ sets this label on that Pod. The value of the label is the name of the Pod being See [Pod Name Label](/docs/concepts/workloads/controllers/statefulset/#pod-name-label) in the StatefulSet topic for more details. -## topology.kubernetes.io/region {#topologykubernetesioregion} +### topology.kubernetes.io/region {#topologykubernetesioregion} Example: @@ -136,7 +136,7 @@ Example: See [topology.kubernetes.io/zone](#topologykubernetesiozone). -## topology.kubernetes.io/zone {#topologykubernetesiozone} +### topology.kubernetes.io/zone {#topologykubernetesiozone} Example: @@ -167,7 +167,7 @@ The scheduler (through the _VolumeZonePredicate_ predicate) also will ensure tha If `PersistentVolumeLabel` does not support automatic labeling of your PersistentVolumes, you should consider adding the labels manually (or adding support for `PersistentVolumeLabel`). With `PersistentVolumeLabel`, the scheduler prevents Pods from mounting volumes in a different zone. If your infrastructure doesn't have this constraint, you don't need to add the zone labels to the volumes at all. -## volume.beta.kubernetes.io/storage-provisioner (deprecated) +### volume.beta.kubernetes.io/storage-provisioner (deprecated) Example: `volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath` @@ -175,13 +175,13 @@ Used on: PersistentVolumeClaim This annotation has been deprecated. -## volume.kubernetes.io/storage-provisioner +### volume.kubernetes.io/storage-provisioner Used on: PersistentVolumeClaim This annotation will be added to dynamic provisioning required PVC. -## node.kubernetes.io/windows-build {#nodekubernetesiowindows-build} +### node.kubernetes.io/windows-build {#nodekubernetesiowindows-build} Example: `node.kubernetes.io/windows-build=10.0.17763` @@ -191,7 +191,7 @@ When the kubelet is running on Microsoft Windows, it automatically labels its no The label's value is in the format "MajorVersion.MinorVersion.BuildNumber". -## service.kubernetes.io/headless {#servicekubernetesioheadless} +### service.kubernetes.io/headless {#servicekubernetesioheadless} Example: `service.kubernetes.io/headless=""` @@ -199,7 +199,7 @@ Used on: Service The control plane adds this label to an Endpoints object when the owning Service is headless. -## kubernetes.io/service-name {#kubernetesioservice-name} +### kubernetes.io/service-name {#kubernetesioservice-name} Example: `kubernetes.io/service-name="nginx"` @@ -207,7 +207,7 @@ Used on: Service Kubernetes uses this label to differentiate multiple Services. Used currently for `ELB`(Elastic Load Balancer) only. -## endpointslice.kubernetes.io/managed-by {#endpointslicekubernetesiomanaged-by} +### endpointslice.kubernetes.io/managed-by {#endpointslicekubernetesiomanaged-by} Example: `endpointslice.kubernetes.io/managed-by="controller"` @@ -215,7 +215,7 @@ Used on: EndpointSlices The label is used to indicate the controller or entity that manages an EndpointSlice. This label aims to enable different EndpointSlice objects to be managed by different controllers or entities within the same cluster. -## endpointslice.kubernetes.io/skip-mirror {#endpointslicekubernetesioskip-mirror} +### endpointslice.kubernetes.io/skip-mirror {#endpointslicekubernetesioskip-mirror} Example: `endpointslice.kubernetes.io/skip-mirror="true"` @@ -223,7 +223,7 @@ Used on: Endpoints The label can be set to `"true"` on an Endpoints resource to indicate that the EndpointSliceMirroring controller should not mirror this resource with EndpointSlices. -## service.kubernetes.io/service-proxy-name {#servicekubernetesioservice-proxy-name} +### service.kubernetes.io/service-proxy-name {#servicekubernetesioservice-proxy-name} Example: `service.kubernetes.io/service-proxy-name="foo-bar"` @@ -231,7 +231,7 @@ Used on: Service The kube-proxy has this label for custom proxy, which delegates service control to custom proxy. -## experimental.windows.kubernetes.io/isolation-type (deprecated) {#experimental-windows-kubernetes-io-isolation-type} +### experimental.windows.kubernetes.io/isolation-type (deprecated) {#experimental-windows-kubernetes-io-isolation-type} Example: `experimental.windows.kubernetes.io/isolation-type: "hyperv"` @@ -244,7 +244,7 @@ You can only set this annotation on Pods that have a single container. Starting from v1.20, this annotation is deprecated. Experimental Hyper-V support was removed in 1.21. {{< /note >}} -## ingressclass.kubernetes.io/is-default-class +### ingressclass.kubernetes.io/is-default-class Example: `ingressclass.kubernetes.io/is-default-class: "true"` @@ -252,13 +252,13 @@ Used on: IngressClass When a single IngressClass resource has this annotation set to `"true"`, new Ingress resource without a class specified will be assigned this default class. -## kubernetes.io/ingress.class (deprecated) +### kubernetes.io/ingress.class (deprecated) {{< note >}} Starting in v1.18, this annotation is deprecated in favor of `spec.ingressClassName`. {{< /note >}} -## storageclass.kubernetes.io/is-default-class +### storageclass.kubernetes.io/is-default-class Example: `storageclass.kubernetes.io/is-default-class=true` @@ -267,7 +267,7 @@ Used on: StorageClass When a single StorageClass resource has this annotation set to `"true"`, new PersistentVolumeClaim resource without a class specified will be assigned this default class. -## alpha.kubernetes.io/provided-node-ip +### alpha.kubernetes.io/provided-node-ip Example: `alpha.kubernetes.io/provided-node-ip: "10.0.0.1"` @@ -277,7 +277,7 @@ The kubelet can set this annotation on a Node to denote its configured IPv4 addr When kubelet is started with the "external" cloud provider, it sets this annotation on the Node to denote an IP address set from the command line flag (`--node-ip`). This IP is verified with the cloud provider as valid by the cloud-controller-manager. -## batch.kubernetes.io/job-completion-index +### batch.kubernetes.io/job-completion-index Example: `batch.kubernetes.io/job-completion-index: "3"` @@ -286,13 +286,13 @@ Used on: Pod The Job controller in the kube-controller-manager sets this annotation for Pods created with Indexed [completion mode](/docs/concepts/workloads/controllers/job/#completion-mode). -## kubectl.kubernetes.io/default-container +### kubectl.kubernetes.io/default-container Example: `kubectl.kubernetes.io/default-container: "front-end-app"` The value of the annotation is the container name that is default for this Pod. For example, `kubectl logs` or `kubectl exec` without `-c` or `--container` flag will use this default container. -## endpoints.kubernetes.io/over-capacity +### endpoints.kubernetes.io/over-capacity Example: `endpoints.kubernetes.io/over-capacity:truncated` @@ -300,7 +300,7 @@ Used on: Endpoints In Kubernetes clusters v1.22 (or later), the Endpoints controller adds this annotation to an Endpoints resource if it has more than 1000 endpoints. The annotation indicates that the Endpoints resource is over capacity and the number of endpoints has been truncated to 1000. -## batch.kubernetes.io/job-tracking +### batch.kubernetes.io/job-tracking Example: `batch.kubernetes.io/job-tracking: ""` @@ -310,7 +310,7 @@ The presence of this annotation on a Job indicates that the control plane is [tracking the Job status using finalizers](/docs/concepts/workloads/controllers/job/#job-tracking-with-finalizers). You should **not** manually add or remove this annotation. -## scheduler.alpha.kubernetes.io/preferAvoidPods (deprecated) {#scheduleralphakubernetesio-preferavoidpods} +### scheduler.alpha.kubernetes.io/preferAvoidPods (deprecated) {#scheduleralphakubernetesio-preferavoidpods} Used on: Nodes @@ -320,61 +320,61 @@ Use [Taints and Tolerations](/docs/concepts/scheduling-eviction/taint-and-tolera **The taints listed below are always used on Nodes** -## node.kubernetes.io/not-ready +### node.kubernetes.io/not-ready Example: `node.kubernetes.io/not-ready:NoExecute` The node controller detects whether a node is ready by monitoring its health and adds or removes this taint accordingly. -## node.kubernetes.io/unreachable +### node.kubernetes.io/unreachable Example: `node.kubernetes.io/unreachable:NoExecute` The node controller adds the taint to a node corresponding to the [NodeCondition](/docs/concepts/architecture/nodes/#condition) `Ready` being `Unknown`. -## node.kubernetes.io/unschedulable +### node.kubernetes.io/unschedulable Example: `node.kubernetes.io/unschedulable:NoSchedule` The taint will be added to a node when initializing the node to avoid race condition. -## node.kubernetes.io/memory-pressure +### node.kubernetes.io/memory-pressure Example: `node.kubernetes.io/memory-pressure:NoSchedule` The kubelet detects memory pressure based on `memory.available` and `allocatableMemory.available` observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. -## node.kubernetes.io/disk-pressure +### node.kubernetes.io/disk-pressure Example: `node.kubernetes.io/disk-pressure:NoSchedule` The kubelet detects disk pressure based on `imagefs.available`, `imagefs.inodesFree`, `nodefs.available` and `nodefs.inodesFree`(Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. -## node.kubernetes.io/network-unavailable +### node.kubernetes.io/network-unavailable Example: `node.kubernetes.io/network-unavailable:NoSchedule` This is initially set by the kubelet when the cloud provider used indicates a requirement for additional network configuration. Only when the route on the cloud is configured properly will the taint be removed by the cloud provider. -## node.kubernetes.io/pid-pressure +### node.kubernetes.io/pid-pressure Example: `node.kubernetes.io/pid-pressure:NoSchedule` The kubelet checks D-value of the size of `/proc/sys/kernel/pid_max` and the PIDs consumed by Kubernetes on a node to get the number of available PIDs that referred to as the `pid.available` metric. The metric is then compared to the corresponding threshold that can be set on the kubelet to determine if the node condition and taint should be added/removed. -## node.cloudprovider.kubernetes.io/uninitialized +### node.cloudprovider.kubernetes.io/uninitialized Example: `node.cloudprovider.kubernetes.io/uninitialized:NoSchedule` Sets this taint on a node to mark it as unusable, when kubelet is started with the "external" cloud provider, until a controller from the cloud-controller-manager initializes this node, and then removes the taint. -## node.cloudprovider.kubernetes.io/shutdown +### node.cloudprovider.kubernetes.io/shutdown Example: `node.cloudprovider.kubernetes.io/shutdown:NoSchedule` If a Node is in a cloud provider specified shutdown state, the Node gets tainted accordingly with `node.cloudprovider.kubernetes.io/shutdown` and the taint effect of `NoSchedule`. -## pod-security.kubernetes.io/enforce +### pod-security.kubernetes.io/enforce Example: `pod-security.kubernetes.io/enforce: baseline` @@ -388,20 +388,20 @@ the requirements outlined in the indicated level. See [Enforcing Pod Security at the Namespace Level](/docs/concepts/security/pod-security-admission) for more information. -## pod-security.kubernetes.io/enforce-version +### pod-security.kubernetes.io/enforce-version Example: `pod-security.kubernetes.io/enforce-version: {{< skew latestVersion >}}` Used on: Namespace Value **must** be `latest` or a valid Kubernetes version in the format `v.`. -This determines the version of the [Pod Security Standard](/docs/concepts/security/pod-security-standards) +This determines the version of the [Pod Security Standard](/docs/concepts/security/pod-security-standards) policies to apply when validating a submitted Pod. See [Enforcing Pod Security at the Namespace Level](/docs/concepts/security/pod-security-admission) for more information. -## pod-security.kubernetes.io/audit +### pod-security.kubernetes.io/audit Example: `pod-security.kubernetes.io/audit: baseline` @@ -415,20 +415,20 @@ the requirements outlined in the indicated level, but adds an audit annotation t See [Enforcing Pod Security at the Namespace Level](/docs/concepts/security/pod-security-admission) for more information. -## pod-security.kubernetes.io/audit-version +### pod-security.kubernetes.io/audit-version Example: `pod-security.kubernetes.io/audit-version: {{< skew latestVersion >}}` Used on: Namespace Value **must** be `latest` or a valid Kubernetes version in the format `v.`. -This determines the version of the [Pod Security Standard](/docs/concepts/security/pod-security-standards) +This determines the version of the [Pod Security Standard](/docs/concepts/security/pod-security-standards) policies to apply when validating a submitted Pod. See [Enforcing Pod Security at the Namespace Level](/docs/concepts/security/pod-security-admission) for more information. -## pod-security.kubernetes.io/warn +### pod-security.kubernetes.io/warn Example: `pod-security.kubernetes.io/warn: baseline` @@ -436,7 +436,7 @@ Used on: Namespace Value **must** be one of `privileged`, `baseline`, or `restricted` which correspond to [Pod Security Standard](/docs/concepts/security/pod-security-standards) levels. Specifically, -the `warn` label does not prevent the creation of a Pod in the labeled Namespace which does not meet the +the `warn` label does not prevent the creation of a Pod in the labeled Namespace which does not meet the requirements outlined in the indicated level, but returns a warning to the user after doing so. Note that warnings are also displayed when creating or updating objects that contain Pod templates, such as Deployments, Jobs, StatefulSets, etc. @@ -444,7 +444,7 @@ such as Deployments, Jobs, StatefulSets, etc. See [Enforcing Pod Security at the Namespace Level](/docs/concepts/security/pod-security-admission) for more information. -## pod-security.kubernetes.io/warn-version +### pod-security.kubernetes.io/warn-version Example: `pod-security.kubernetes.io/warn-version: {{< skew latestVersion >}}` @@ -458,7 +458,7 @@ or updating objects that contain Pod templates, such as Deployments, Jobs, State See [Enforcing Pod Security at the Namespace Level](/docs/concepts/security/pod-security-admission) for more information. -## seccomp.security.alpha.kubernetes.io/pod (deprecated) {#seccomp-security-alpha-kubernetes-io-pod} +### seccomp.security.alpha.kubernetes.io/pod (deprecated) {#seccomp-security-alpha-kubernetes-io-pod} This annotation has been deprecated since Kubernetes v1.19 and will become non-functional in v1.25. To specify security settings for a Pod, include the `securityContext` field in the Pod specification. @@ -466,10 +466,74 @@ The [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1 When you [specify the security context for a Pod](/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod), the settings you specify apply to all containers in that Pod. -## container.seccomp.security.alpha.kubernetes.io/[NAME] {#container-seccomp-security-alpha-kubernetes-io} +### container.seccomp.security.alpha.kubernetes.io/[NAME] {#container-seccomp-security-alpha-kubernetes-io} This annotation has been deprecated since Kubernetes v1.19 and will become non-functional in v1.25. The tutorial [Restrict a Container's Syscalls with seccomp](/docs/tutorials/clusters/seccomp/) takes you through the steps you follow to apply a seccomp profile to a Pod or to one of its containers. That tutorial covers the supported mechanism for configuring seccomp in Kubernetes, based on setting `securityContext` within the Pod's `.spec`. + +## kubeadm + +### kubeadm.alpha.kubernetes.io/cri-socket + +Example: `kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/container.sock` + +Used on: Node + +Annotation that kubeadm uses to preserve the CRI socket information given to kubeadm at `init`/`join` time for later use. +kubeadm annotates the Node object with this information. The annotation remains "alpha", since ideally this should +be a field in KubeletConfiguration instead. + +### kubeadm.kubernetes.io/etcd.advertise-client-urls + +Example: `kubeadm.kubernetes.io/etcd.advertise-client-urls: https://172.17.0.18:2379` + +Used on: Pod + +Annotation that kubeadm places on locally managed etcd pods to keep track of a list of URLs where etcd clients +should connect to. This is used mainly for etcd cluster health check purposes. + +### kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint + +Example: `kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: https//172.17.0.18:6443` + +Used on: Pod + +Annotation that kubeadm places on locally managed kube-apiserver pods to keep track of the exposed advertise +address/port endpoint for that API server instance. + +### kubeadm.kubernetes.io/component-config.hash + +Used on: ConfigMap + +Example: `kubeadm.kubernetes.io/component-config.hash: 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae` + +Annotation that kubeadm places on ConfigMaps that it manages for configuring components. It contains a hash (SHA-256) +used to determine if the user has applied settings different from the kubeadm defaults for a particular component. + +### node-role.kubernetes.io/control-plane + +Used on: Node + +Label that kubeadm applies on the control plane nodes that it manages. + +### node-role.kubernetes.io/control-plane + +Used on: Node + +Example: `node-role.kubernetes.io/control-plane:NoSchedule` + +Taint that kubeadm applies on control plane nodes to allow only critical workloads to schedule on them. + +### node-role.kubernetes.io/master + +Used on: Node + +Example: `node-role.kubernetes.io/master:NoSchedule` + +Taint that kubeadm applies on control plane nodes to allow only critical workloads to schedule on them. + +{{< note >}} Starting in v1.20, this taint is deprecated in favor of `node-role.kubernetes.io/control-plane` +and will be removed in v1.25.{{< /note >}}