From 3292ae8e09657100bd3b7b47c5b3effd1f04c5f9 Mon Sep 17 00:00:00 2001 From: xuzhenglun Date: Wed, 22 Mar 2023 10:06:38 +0800 Subject: [PATCH] add blog document for feature ServiceNodePortStaticSubrange --- ...-nodeport-dynamic-and-static-allocation.md | 173 ++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 content/en/blog/_posts/2023-05-11-nodeport-dynamic-and-static-allocation.md diff --git a/content/en/blog/_posts/2023-05-11-nodeport-dynamic-and-static-allocation.md b/content/en/blog/_posts/2023-05-11-nodeport-dynamic-and-static-allocation.md new file mode 100644 index 0000000000..fde6ed1c1d --- /dev/null +++ b/content/en/blog/_posts/2023-05-11-nodeport-dynamic-and-static-allocation.md @@ -0,0 +1,173 @@ +--- +layout: blog +title: "Kubernetes 1.27: Avoid Collisions Assigning Ports to NodePort Services" +date: 2023-05-11 +slug: nodeport-dynamic-and-static-allocation +--- + +**Author:** Xu Zhenglun (Alibaba) + +In Kubernetes, a Service can be used to provide a unified traffic endpoint for +applications running on a set of Pods. Clients can use the virtual IP address (or _VIP_) provided +by the Service for access, and Kubernetes provides load balancing for traffic accessing +different back-end Pods, but a ClusterIP type of Service is limited to providing access to +nodes within the cluster, while traffic from outside the cluster cannot be routed. +One way to solve this problem is to use a `type: NodePort` Service, which sets up a mapping +to a specific port of all nodes in the cluster, thus redirecting traffic from the +outside to the inside of the cluster. + +## How Kubernetes allocates node ports to Services? + +When a `type: NodePort` Service is created, its corresponding port(s) are allocated in one +of two ways: + +- **Dynamic** : If the Service type is `NodePort` and you do not set a `nodePort` + value explicitly in the `spec` for that Service, the Kubernetes control plane will + automatically allocate an unused port to it at creation time. + +- **Static** : In addition to the dynamic auto-assignment described above, you can also + explicitly assign a port that is within the nodeport port range configuration. + +The value of `nodePort` that you manually assign must be unique across the whole cluster. +Attempting to create a Service of `type: NodePort` where you explicitly specify a node port that +was already allocated results in an error. + +## Why do you need to reserve ports of NodePort Service? +Sometimes, you may want to have a NodePort Service running on well-known ports +so that other components and users inside o r outside the cluster can use them. + +In some complex cluster deployments with a mix of Kubernetes nodes and other servers on the same network, +it may be necessary to use some pre-defined ports for communication. In particular, some fundamental +components cannot rely on the VIPs that back `type: LoadBalancer` Services +because the virtual IP address mapping implementation for that cluster also relies on +these foundational components. + +Now suppose you need to expose a Minio object storage service on Kubernetes to clients +running outside the Kubernetes cluster, and the agreed port is `30009`, we need to +create a Service as follows: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: minio +spec: + ports: + - name: api + nodePort: 30009 + port: 9000 + protocol: TCP + targetPort: 9000 + selector: + app: minio + type: NodePort +``` + +However, as mentioned before, if the port (30009) required for the `minio` Service is not reserved, +and another `type: NodePort` (or possibly `type: LoadBalancer`) Service is created and dynamically +allocated before or concurrently with the `minio` Service, TCP port 30009 might be allocated to that +other Service; if so, creation of the `minio` Service will fail due to a node port collision. + +## How can you avoid NodePort Service port conflicts? +Kubernetes 1.24 introduced changes for `type: ClusterIP` Services, dividing the CIDR range for cluster +IP addresses into two blocks that use different allocation policies to [reduce the risk of conflicts](/docs/reference/networking/virtual-ips/#avoiding-collisions). +In Kubernetes 1.27, as an alpha feature, you can adopt a similar policy for `type: NodePort` Servies. +You can enable a new [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +`ServiceNodePortStaticSubrange`. Turning this on allows you to use a different port allocation strategy +for `type: NodePort` Services, and reduce the risk of collision. + +The port range for `NodePort` will be divided, based on the formula `min(max(16, nodeport-size / 32), 128)`. +The outcome of the formula will be a number between 16 and 128, with a step size that increases as the +size of the nodeport range increases. The outcome of the formula determine that the size of static port +range. When the port range is less than 16, the size of static port range will be set to 0, +which means that all ports will be dynamically allocated. + +Dynamic port assignment will use the upper band by default, once this has been exhausted it will use the lower range. +This will allow users to use static allocations on the lower band with a low risk of collision. + +## Examples + +### default range: 30000-32767 +| Range properties | Values | +|-------------------------|-------------------------------------------------------| +| service-node-port-range | 30000-32767 | +| Band Offset |   `min(max(16, 2768/32), 128)`
= `min(max(16, 86), 128)`
= `min(86, 128)`
= 86 | +| Static band start | 30000 | +| Static band end | 30085 | +| Dynamic band start | 30086 | +| Dynamic band end | 32767 | + +{{< mermaid >}} +pie showData + title 30000-32767 + "Static" : 86 + "Dynamic" : 2682 +{{< /mermaid >}} + +### very small range: 30000-30015 +| Range properties | Values | +|-------------------------|-------------------------------------------------------| +| service-node-port-range | 30000-30015 | +| Band Offset | 0 | +| Static band start | - | +| Static band end | - | +| Dynamic band start | 30000 | +| Dynamic band end | 30015 | + +{{< mermaid >}} +pie showData + title 30000-30015 + "Static" : 0 + "Dynamic" : 16 +{{< /mermaid >}} + +### small(lower boundary) range: 30000-30127 +| Range properties | Values | +|-------------------------|-------------------------------------------------------| +| service-node-port-range | 30000-30127 | +| Band Offset |   `min(max(16, 128/32), 128)`
= `min(max(16, 4), 128)`
= `min(16, 128)`
= 16 | +| Static band start | 30000 | +| Static band end | 30015 | +| Dynamic band start | 30016 | +| Dynamic band end | 30127 | + +{{< mermaid >}} +pie showData + title 30000-30127 + "Static" : 16 + "Dynamic" : 112 +{{< /mermaid >}} + +### large(upper boundary) range: 30000-34095 +| Range properties | Values | +|-------------------------|-------------------------------------------------------| +| service-node-port-range | 30000-34095 | +| Band Offset |   `min(max(16, 4096/32), 128)`
= `min(max(16, 128), 128)`
= `min(128, 128)`
= 128 | +| Static band start | 30000 | +| Static band end | 30127 | +| Dynamic band start | 30128 | +| Dynamic band end | 34095 | + +{{< mermaid >}} +pie showData + title 30000-34095 + "Static" : 128 + "Dynamic" : 3968 +{{< /mermaid >}} + +### very large range: 30000-38191 +| Range properties | Values | +|-------------------------|-------------------------------------------------------| +| service-node-port-range | 30000-38191 | +| Band Offset |   `min(max(16, 8192/32), 128)`
= `min(max(16, 256), 128)`
= `min(256, 128)`
= 128 | +| Static band start | 30000 | +| Static band end | 30127 | +| Dynamic band start | 30128 | +| Dynamic band end | 38191 | + +{{< mermaid >}} +pie showData + title 30000-38191 + "Static" : 128 + "Dynamic" : 8064 +{{< /mermaid >}} \ No newline at end of file