From bb9c7d9d2457070d0d794a50bfeb07422fa86c98 Mon Sep 17 00:00:00 2001 From: Richa Banker Date: Sun, 23 Jul 2023 15:38:14 -0700 Subject: [PATCH] Add mvp documentation Co-authored-by: Tim Bannister Co-authored-by: Joe Betz --- .../architecture/mixed-version-proxy.md | 84 +++++++++++++++++++ .../feature-gates.md | 4 + 2 files changed, 88 insertions(+) create mode 100644 content/en/docs/concepts/architecture/mixed-version-proxy.md diff --git a/content/en/docs/concepts/architecture/mixed-version-proxy.md b/content/en/docs/concepts/architecture/mixed-version-proxy.md new file mode 100644 index 0000000000..fd07b140c0 --- /dev/null +++ b/content/en/docs/concepts/architecture/mixed-version-proxy.md @@ -0,0 +1,84 @@ +--- +reviewers: +- jpbetz +title: Mixed Version Proxy +content_type: concept +weight: 220 +--- + + +{{< feature-state state="alpha" for_k8s_version="v1.28" >}} + +Kubernetes {{< skew currentVersion >}} includes an alpha feature that lets a +{{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}} +proxy a resource requests to other _peer_ API servers. This is useful when there are multiple +API servers running different versions of Kubernetes in one cluster (for example, during a long-lived +rollout to a new release of Kubernetes). + +This enables cluster administrators to configure highly available clusters that can be upgraded +more safely, by directing resource requests (made during the upgrade) to the correct kube-apiserver. +That proxying prevents users from seeing unexpected 404 Not Found errors that stem +from the upgrade process. + +This mechanism is called the _Mixed Version Proxy_. + +## Enabling the Mixed Version Proxy +Ensure that `UnknownVersionInteroperabilityProxy` [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) +is enabled when you start the {{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}}: + +```shell +kube-apiserver \ +--feature-gates=UnknownVersionInteroperabilityProxy=true \ +# required command line arguments for this feature +--peer-ca-file= +--proxy-client-cert-file=, +--proxy-client-key-file=, +--requestheader-client-ca-file=, +# requestheader-allowed-names can be set to blank to allow any Common Name +--requestheader-allowed-names=, + +# optional flags for this feature +--peer-advertise-ip=`IP of this kube-apiserver that should be used by peers to proxy requests` +--peer-advertise-port=`port of this kube-apiserver that should be used by peers to proxy requests` + +# …and other flags as usual +``` + +### Proxy transport and authentication between API servers {#transport-and-authn} + +* The source kube-apiserver reuses the [existing APIserver client authentication flags](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#kubernetes-apiserver-client-authentication) `--proxy-client-cert-file` and `--proxy-client-key-file` to present its identity that will be verified by its peer (the destination kube-apiserver). The destination API server verifies that peer connection based on the configuration you specify using the `--requestheader-client-ca-file` command line argument. + +* To authenticate the destination server's serving certs, you must configure a certificate authority bundle by specifying the `--peer-ca-file` command line argument to the **source** API server. + +### Configuration for peer API server connectivity + +To set the network location of a kube-apiserver that peers will use to proxy requests, use the +`--peer-advertise-ip` and `--peer-advertise-port` command line arguments to kube-apiserver or specify +these fields in the API server configuration file. +If these flags are unspecified, peers will use the value from either `--advertise-address` or +`--bind-address` command line argument to the kube-apiserver. If those too, are unset, the host's default interface is used. + +## Mixed version proxying + +When you enable mixed version proxying, the [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) +loads a special filter that does the following: + +* When a resource request reaches an API server that cannot serve that API (either because it is at a version pre-dating the introduction of the API or the API is turned off on the API server) the API server attempts to send the request to a peer API server that can serve the requested API. It does so by identifying API groups / versions / resources that the local server doesn't recognise, and tries to proxy those requests to a peer API server that is capable of handling the request. +* If the peer API server fails to respond, the _source_ API server responds with 503("Service Unavailable") error. + +### How it works under the hood + +When an API Server receives a resource request, it first checks which API servers can serve the requested resource. This check happens using the internal [`StorageVersion` API]. + +* If the resource is known to the API server that received the request (ex: `GET /api/v1/pods/some-pod`), the request is handled locally. + +* If there is no internal `StorageVersion` object found for the requested resource (ex: `GET /my-api/v1/my-resource`) and the configured APIService specifies proxying to an extension API server, that proxying happens following the usual +[flow](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) for +extension APIs. + +* If a valid internal `StorageVersion` object is found for the requested resource (ex: `GET /batch/v1/jobs`) and the API server trying to handle the request (the _handling API server_) has the `batch` API disabled, then the _handling API server_fetches the peer API servers that do serve the relevant API group / version / resource (`api/v1/batch` in this case) using the information in the fetched `StorageVersion` object. The _handling API server_ then proxies the request to one of the matching peer kube-apiservers that are aware of the requested resource. + * If there is no peer known for that API group / version / resource, the handling API server passes the request to its own handler chain which should eventually return a 404("Not Found") response. + * If the handling API server has identified and selected a peer API server, but that peer fails + to respond (for reasons such as network connectivity issues, or a data race between the request + being received and a controller registering the peer's info into the control plane), then the handling + API server responds with a 503 (“Service Unavailable”) error. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 1061a15f5e..cab25fff93 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -210,6 +210,7 @@ For a reference to old feature gates that are removed, please refer to | `TopologyManagerPolicyAlphaOptions` | `false` | Alpha | 1.26 | | | `TopologyManagerPolicyBetaOptions` | `false` | Beta | 1.26 | | | `TopologyManagerPolicyOptions` | `false` | Alpha | 1.26 | | +| `UnknownVersionInteroperabilityProxy` | `false` | Alpha | 1.28 | | | `UserNamespacesStatelessPodsSupport` | `false` | Alpha | 1.25 | | | `ValidatingAdmissionPolicy` | `false` | Alpha | 1.26 | | | `VolumeCapacityPriority` | `false` | Alpha | 1.21 | - | @@ -732,6 +733,9 @@ Each feature gate is designed for enabling/disabling a specific feature: This feature gate guards *a group* of topology manager options whose quality level is beta. This feature gate will never graduate to stable. - `TopologyManagerPolicyOptions`: Allow fine-tuning of topology manager policies, +- `UnknownVersionInteroperabilityProxy`: Proxy resource requests to the correct peer kube-apiserver when + multiple kube-apiservers exist at varied versions. + See [Mixed version proxy](/docs/concepts/architecture/mixed-version-proxy/) for more information. - `UserNamespacesStatelessPodsSupport`: Enable user namespace support for stateless Pods. - `ValidatingAdmissionPolicy`: Enable [ValidatingAdmissionPolicy](/docs/reference/access-authn-authz/validating-admission-policy/) support for CEL validations be used in Admission Control. - `VolumeCapacityPriority`: Enable support for prioritizing nodes in different