Merge pull request #29783 from Nordix/kpng-1
Added blog post kpng-specialized-proxierspull/30048/head
Kubernetes Prow Robot
GitHub
commit
873a25adf7
|
|
@ -0,0 +1,241 @@
|
|||
---
|
||||
layout: blog
|
||||
title: "Use KPNG to Write Specialized kube-proxiers"
|
||||
date: 2021-10-18
|
||||
slug: use-kpng-to-write-specialized-kube-proxiers
|
||||
---
|
||||
|
||||
Authors: Lars Ekman, Ericsson
|
||||
|
||||
The post will show you how to create a specialized service kube-proxy
|
||||
style network proxier using Kubernetes Proxy NG
|
||||
[kpng](https://github.com/kubernetes-sigs/kpng) without interfering
|
||||
with the existing kube-proxy. The kpng project aims at renewing the
|
||||
the default Kubernetes Service implementation, the "kube-proxy". An
|
||||
important feature of kpng is that it can be used as a library to
|
||||
create proxiers outside K8s. While this is useful for CNI-plugins that
|
||||
replaces the kube-proxy it also opens the possibility for anyone to
|
||||
create a proxier for a special purpose.
|
||||
|
||||
|
||||
## Define a service that uses a specialized proxier
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: kpng-example
|
||||
labels:
|
||||
service.kubernetes.io/service-proxy-name: kpng-example
|
||||
spec:
|
||||
clusterIP: None
|
||||
ipFamilyPolicy: RequireDualStack
|
||||
externalIPs:
|
||||
- 10.0.0.55
|
||||
- 1000::55
|
||||
selector:
|
||||
app: kpng-alpine
|
||||
ports:
|
||||
- port: 6000
|
||||
```
|
||||
|
||||
If the `service.kubernetes.io/service-proxy-name` label is defined the
|
||||
`kube-proxy` will ignore the service. A custom controller can watch
|
||||
services with the label set to it's own name, "kpng-example" in
|
||||
this example, and setup specialized load-balancing.
|
||||
|
||||
The `service.kubernetes.io/service-proxy-name` label is [not
|
||||
new](https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioservice-proxy-name),
|
||||
but so far is has been quite hard to write a specialized proxier.
|
||||
|
||||
The common use for a specialized proxier is assumed to be handling
|
||||
external traffic for some use-case not supported by K8s. In that
|
||||
case `ClusterIP` is not needed, so we use a "headless" service in this
|
||||
example.
|
||||
|
||||
|
||||
## Specialized proxier using kpng
|
||||
|
||||
A [kpng](https://github.com/kubernetes-sigs/kpng) based proxier
|
||||
consists of the `kpng` controller handling all the K8s api related
|
||||
functions, and a "backend" implementing the load-balancing. The
|
||||
backend can be linked with the `kpng` controller binary or be a
|
||||
separate program communicating with the controller using gRPC.
|
||||
|
||||
```
|
||||
kpng kube --service-proxy-name=kpng-example to-api
|
||||
```
|
||||
|
||||
This starts the `kpng` controller and tell it to watch only services
|
||||
with the "kpng-example" service proxy name. The "to-api" parameter
|
||||
will open a gRPC server for backends.
|
||||
|
||||
You can test this yourself outside your cluster. Please see the example
|
||||
below.
|
||||
|
||||
Now we start a backend that simply prints the updates from the
|
||||
controller.
|
||||
|
||||
```
|
||||
$ kubectl apply -f kpng-example.yaml
|
||||
$ kpng-json | jq # (this is the backend)
|
||||
{
|
||||
"Service": {
|
||||
"Namespace": "default",
|
||||
"Name": "kpng-example",
|
||||
"Type": "ClusterIP",
|
||||
"IPs": {
|
||||
"ClusterIPs": {},
|
||||
"ExternalIPs": {
|
||||
"V4": [
|
||||
"10.0.0.55"
|
||||
],
|
||||
"V6": [
|
||||
"1000::55"
|
||||
]
|
||||
},
|
||||
"Headless": true
|
||||
},
|
||||
"Ports": [
|
||||
{
|
||||
"Protocol": 1,
|
||||
"Port": 6000,
|
||||
"TargetPort": 6000
|
||||
}
|
||||
]
|
||||
},
|
||||
"Endpoints": [
|
||||
{
|
||||
"IPs": {
|
||||
"V6": [
|
||||
"1100::202"
|
||||
]
|
||||
},
|
||||
"Local": true
|
||||
},
|
||||
{
|
||||
"IPs": {
|
||||
"V4": [
|
||||
"11.0.2.2"
|
||||
]
|
||||
},
|
||||
"Local": true
|
||||
},
|
||||
{
|
||||
"IPs": {
|
||||
"V4": [
|
||||
"11.0.1.2"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"IPs": {
|
||||
"V6": [
|
||||
"1100::102"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A real backend would use some mechanism to load-balance traffic from
|
||||
the external IPs to the endpoints.
|
||||
|
||||
|
||||
|
||||
## Writing a backend
|
||||
|
||||
The `kpng-json` backend looks like this:
|
||||
|
||||
```go
|
||||
package main
|
||||
import (
|
||||
"os"
|
||||
"encoding/json"
|
||||
"sigs.k8s.io/kpng/client"
|
||||
)
|
||||
func main() {
|
||||
client.Run(jsonPrint)
|
||||
}
|
||||
func jsonPrint(items []*client.ServiceEndpoints) {
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
for _, item := range items {
|
||||
_ = enc.Encode(item)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(yes, that is the entire program)
|
||||
|
||||
A real backend would of course be much more complex, but this
|
||||
illustrates how `kpng` let you focus on load-balancing.
|
||||
|
||||
You can have several backends connected to a `kpng` controller, so
|
||||
during development or debug it can be useful to let something like the
|
||||
`kpng-json` backend run in parallel with your real backend.
|
||||
|
||||
|
||||
## Example
|
||||
|
||||
|
||||
The complete example can be found [here](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
|
||||
|
||||
As an example we implement an "all-ip" backend. It direct all traffic
|
||||
for the externalIPs to a local endpoint, regardless of ports and upper
|
||||
layer protocols. There is a
|
||||
[KEP](https://github.com/kubernetes/enhancements/pull/2611) for this
|
||||
function and this example is a much simplified version.
|
||||
|
||||
To direct all traffic from an external address to a local POD [only
|
||||
one iptables rule is
|
||||
needed](https://github.com/kubernetes/enhancements/pull/2611#issuecomment-895061013),
|
||||
for instance;
|
||||
|
||||
```
|
||||
ip6tables -t nat -A PREROUTING -d 1000::55/128 -j DNAT --to-destination 1100::202
|
||||
```
|
||||
|
||||
As you can see the addresses are in the call to the backend and all it
|
||||
have to do is:
|
||||
|
||||
* Extract the addresses with `Local: true`
|
||||
* Setup iptables rules for the `ExternalIPs`
|
||||
|
||||
A script doing that may look like:
|
||||
|
||||
```
|
||||
xip=$(cat /tmp/out | jq -r .Service.IPs.ExternalIPs.V6[0])
|
||||
podip=$(cat /tmp/out | jq -r '.Endpoints[]|select(.Local == true)|select(.IPs.V6 != null)|.IPs.V6[0]')
|
||||
ip6tables -t nat -A PREROUTING -d $xip/128 -j DNAT --to-destination $podip
|
||||
```
|
||||
|
||||
Assuming the json output above is stored in `/tmp/out` ([jq](https://stedolan.github.io/jq/) is an *awesome* program!).
|
||||
|
||||
|
||||
As this is an example we make it really simple for ourselves by using
|
||||
a minor variation of the `kpng-json` backend above. Instead of just
|
||||
printing, a program is called and the json output is passed as `stdin`
|
||||
to that program. The backend can be tested stand-alone:
|
||||
|
||||
```
|
||||
CALLOUT=jq kpng-callout
|
||||
```
|
||||
|
||||
Where `jq` can be replaced with your own program or script. A script
|
||||
may look like the example above. For more info and the complete
|
||||
example please see [https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
While [kpng](https://github.com/kubernetes-sigs/kpng) is in early
|
||||
stage of development this post wants to show how you may build your
|
||||
own specialized K8s proxiers in the future. The only thing your
|
||||
applications need to do is to add the
|
||||
`service.kubernetes.io/service-proxy-name` label in the service
|
||||
manifest.
|
||||
|
||||
It is a tedious process to get new features into the `kube-proxy` and
|
||||
it is not unlikely that they will be rejected, so to write a
|
||||
specialized proxier may be the only option.
|
||||
Loading…
Reference in New Issue