From 99dedbabf9094f47ac5e9b30d96159fb429befa5 Mon Sep 17 00:00:00 2001 From: Rob Scott Date: Fri, 17 Mar 2023 02:01:38 +0000 Subject: [PATCH] Updating documentation for Topology Aware Routing in 1.27 --- .../services-networking/service-topology.md | 4 +- ...are-hints.md => topology-aware-routing.md} | 91 +++++++++++++------ static/_redirects | 1 + 3 files changed, 68 insertions(+), 28 deletions(-) rename content/en/docs/concepts/services-networking/{topology-aware-hints.md => topology-aware-routing.md} (62%) diff --git a/content/en/docs/concepts/services-networking/service-topology.md b/content/en/docs/concepts/services-networking/service-topology.md index b833ff6e3b..081a58cc85 100644 --- a/content/en/docs/concepts/services-networking/service-topology.md +++ b/content/en/docs/concepts/services-networking/service-topology.md @@ -16,8 +16,8 @@ weight: 150 This feature, specifically the alpha `topologyKeys` API, is deprecated since Kubernetes v1.21. -[Topology Aware Hints](/docs/concepts/services-networking/topology-aware-hints/), -introduced in Kubernetes v1.21, provide similar functionality. +[Topology Aware Routing](/docs/concepts/services-networking/topology-aware-routing/), +introduced in Kubernetes v1.21, provides similar functionality. {{}} _Service Topology_ enables a service to route traffic based upon the Node diff --git a/content/en/docs/concepts/services-networking/topology-aware-hints.md b/content/en/docs/concepts/services-networking/topology-aware-routing.md similarity index 62% rename from content/en/docs/concepts/services-networking/topology-aware-hints.md rename to content/en/docs/concepts/services-networking/topology-aware-routing.md index 7a6d212476..1977f199d0 100644 --- a/content/en/docs/concepts/services-networking/topology-aware-hints.md +++ b/content/en/docs/concepts/services-networking/topology-aware-routing.md @@ -1,11 +1,11 @@ --- reviewers: - robscott -title: Topology Aware Hints +title: Topology Aware Routing content_type: concept weight: 100 description: >- - _Topology Aware Hints_ provides a mechanism to help keep network traffic within the zone + _Topology Aware Routing_ provides a mechanism to help keep network traffic within the zone where it originated. Preferring same-zone traffic between Pods in your cluster can help with reliability, performance (network latency and throughput), or cost. --- @@ -15,45 +15,68 @@ description: >- {{< feature-state for_k8s_version="v1.23" state="beta" >}} -_Topology Aware Hints_ enable topology aware routing by including suggestions -for how clients should consume endpoints. This approach adds metadata to enable -consumers of EndpointSlice (or Endpoints) objects, so that traffic to -those network endpoints can be routed closer to where it originated. +{{< note >}} +Prior to Kubernetes 1.27, this feature was known as _Topology Aware Hints_. +{{}} -For example, you can route traffic within a locality to reduce -costs, or to improve network performance. +_Topology Aware Routing_ adjusts routing behavior to prefer keeping traffic in +the zone it originated from. In some cases this can help reduce costs or improve +network performance. ## Motivation Kubernetes clusters are increasingly deployed in multi-zone environments. -_Topology Aware Hints_ provides a mechanism to help keep traffic within the zone -it originated from. This concept is commonly referred to as "Topology Aware -Routing". When calculating the endpoints for a {{< glossary_tooltip term_id="Service" >}}, -the EndpointSlice controller considers the topology (region and zone) of each endpoint -and populates the hints field to allocate it to a zone. -Cluster components such as the {{< glossary_tooltip term_id="kube-proxy" text="kube-proxy" >}} -can then consume those hints, and use them to influence how the traffic is routed -(favoring topologically closer endpoints). +_Topology Aware Routing_ provides a mechanism to help keep traffic within the +zone it originated from. When calculating the endpoints for a {{< +glossary_tooltip term_id="Service" >}}, the EndpointSlice controller considers +the topology (region and zone) of each endpoint and populates the hints field to +allocate it to a zone. Cluster components such as {{< glossary_tooltip +term_id="kube-proxy" text="kube-proxy" >}} can then consume those hints, and use +them to influence how the traffic is routed (favoring topologically closer +endpoints). -## Using Topology Aware Hints +## Enabling Topology Aware Routing -You can activate Topology Aware Hints for a Service by setting the -`service.kubernetes.io/topology-aware-hints` annotation to `auto`. This tells -the EndpointSlice controller to set topology hints if it is deemed safe. -Importantly, this does not guarantee that hints will always be set. +{{< note >}} +Prior to Kubernetes 1.27, this behavior was controlled using the +`service.kubernetes.io/topology-aware-hints` annotation. +{{}} -## How it works {#implementation} +You can enable Topology Aware Routing for a Service by setting the +`service.kubernetes.io/topology-mode` annotation to `Auto`. When there are +enough endpoints available in each zone, Topology Hints will be populated on +EndpointSlices to allocate individual endpoints to specific zones, resulting in +traffic being routed closer to where it originated from. -The functionality enabling this feature is split into two components: The -EndpointSlice controller and the kube-proxy. This section provides a high level overview -of how each component implements this feature. +## When it works best + +This feature works best when: + +### 1. Incoming traffic is evenly distributed + +If a large proportion of traffic is originating from a single zone, that traffic +could overload the subset of endpoints that have been allocated to that zone. +This feature is not recommended when incoming traffic is expected to originate +from a single zone. + +### 2. The Service has 3 or more endpoints per zone {#three-or-more-endpoints-per-zone} +In a three zone cluster, this means 9 or more endpoints. If there are fewer than +3 endpoints per zone, there is a high (≈50%) probability that the EndpointSlice +controller will not be able to allocate endpoints evenly and instead will fall +back to the default cluster-wide routing approach. + +## How It Works + +The "Auto" heuristic attempts to proportionally allocate a number of endpoints +to each zone. Note that this heuristic works best for Services that have a +significant number of endpoints. ### EndpointSlice controller {#implementation-control-plane} The EndpointSlice controller is responsible for setting hints on EndpointSlices -when this feature is enabled. The controller allocates a proportional amount of +when this heuristic is enabled. The controller allocates a proportional amount of endpoints to each zone. This proportion is based on the [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) CPU cores for nodes running in that zone. For example, if one zone had 2 CPU @@ -145,6 +168,11 @@ zone. proportions of each zone. This could have unintended consequences if a large portion of nodes are unready. +* The EndpointSlice controller ignores nodes with the + `node-role.kubernetes.io/control-plane` or `node-role.kubernetes.io/master` + label set. This could be problematic if workloads are also running on those + nodes. + * The EndpointSlice controller does not take into account {{< glossary_tooltip text="tolerations" term_id="toleration" >}} when deploying or calculating the proportions of each zone. If the Pods backing a Service are limited to a @@ -157,6 +185,17 @@ zone. either not picking up on this event, or newly added pods starting in a different zone. + +## Custom heuristics + +Kubernetes is deployed in many different ways, there is no single heuristic for +allocating endpoints to zones will work for every use case. A key goal of this +feature is to enable custom heuristics to be developed if the built in heuristic +does not work for your use case. The first steps to enable custom heuristics +were included in the 1.27 release. This is a limited implementation that may not +yet cover some relevant and plausible situations. + + ## {{% heading "whatsnext" %}} * Follow the [Connecting Applications with Services](/docs/tutorials/services/connect-applications-service/) tutorial diff --git a/static/_redirects b/static/_redirects index f4bd325090..b720f758cb 100644 --- a/static/_redirects +++ b/static/_redirects @@ -94,6 +94,7 @@ /docs/concepts/service-catalog/ /docs/concepts/extend-kubernetes/service-catalog/ 301 /docs/concepts/services-networking/networkpolicies/ /docs/concepts/services-networking/network-policies/ 301 /docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/ /docs/tasks/network/customize-hosts-file-for-pods/ 301 +/docs/concepts/services-networking/topology-aware-hints/ /docs/concepts/services-networking/topology-aware-routing/ 302 /docs/concepts/storage/etcd-store-api-object/ /docs/tasks/administer-cluster/configure-upgrade-etcd/ 301 /docs/concepts/storage/volumes/emptyDirapiVersion/ /docs/concepts/storage/volumes/#emptydir/ 301 /docs/concepts/tools/kubectl/object-management-overview/ /docs/concepts/overview/object-management-kubectl/overview/ 301