blog: health-checking-grpc (#10393)

* blog: health-checking-grpc

date in the markdown and file/dir names will be updated later

Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

* Update 2018-09-30-health-checking-grpc.md

* Update and rename 2018-09-30-health-checking-grpc.md to 2018-10-01-health-checking-grpc.md

* Update 2018-10-01-health-checking-grpc.md

* Update 2018-10-01-health-checking-grpc.md
pull/9591/merge
Ahmet Alp Balkan 2018-10-01 09:33:49 -07:00 committed by k8s-ci-robot
parent 54ba69b5fb
commit b0de40f874
3 changed files with 97 additions and 0 deletions

View File

@ -0,0 +1,97 @@
---
layout: blog
title: 'Health checking gRPC servers on Kubernetes'
date: 2018-10-01
---
**Author**: [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google)
[gRPC](https://grpc.io) is on its way to becoming the lingua franca for
communication between cloud-native microservices. If you are deploying gRPC
applications to Kubernetes today, you may be wondering about the best way to
configure health checks. In this article, we will talk about
[grpc-health-probe](https://github.com/grpc-ecosystem/grpc-health-probe/), a
Kubernetes-native way to health check gRPC apps.
If you're unfamiliar, Kubernetes [health
checks](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
(liveness and readiness probes) is what's keeping your applications available
while you're sleeping. They detect unresponsive pods, mark them unhealthy, and
cause these pods to be restarted or rescheduled.
Kubernetes [does not
support](https://github.com/kubernetes/kubernetes/issues/21493) gRPC health
checks natively. This leaves the gRPC developers with the following three
approaches when they deploy to Kubernetes:
[![options for health checking grpc on kubernetes today](/images/blog/2019-09-30-health-checking-grpc/options.png)](/images/blog/2019-09-30-health-checking-grpc/options.png)
1. **httpGet probe:** Cannot be natively used with gRPC. You need to refactor
your app to serve both gRPC and HTTP/1.1 protocols (on different port
numbers).
2. **tcpSocket probe:** Opening a socket to gRPC server is not meaningful,
since it cannot read the response body.
3. **exec probe:** This invokes a program in a container's ecosystem
periodically. In the case of gRPC, this means you implement a health RPC
yourself, then write and ship a client tool with your container.
Can we do better? Absolutely.
## Introducing “grpc-health-probe”
To standardize the "exec probe" approach mentioned above, we need:
- a **standard** health check "protocol" that can be implemented in any gRPC
server easily.
- a **standard** health check "tool" that can query the health protocol easily.
Thankfully, gRPC has a [standard health checking
protocol](https://github.com/grpc/grpc/blob/v1.15.0/doc/health-checking.md). It
can be used easily from any language. Generated code and the utilities for
setting the health status are shipped in nearly all language implementations of
gRPC.
If you
[implement](https://github.com/grpc/grpc/blob/v1.15.0/src/proto/grpc/health/v1/health.proto)
this health check protocol in your gRPC apps, you can then use a standard/common
tool to invoke this `Check()` method to determine server status.
The next thing you need is the "standard tool", and it's the
[**grpc-health-probe**](https://github.com/grpc-ecosystem/grpc-health-probe/).
<a href='/images/blog/2019-09-30-health-checking-grpc/grpc_health_probe.png'>
<img width="768" title='grpc-health-probe on kubernetes'
src='/images/blog/2019-09-30-health-checking-grpc/grpc_health_probe.png'/>
</a>
With this tool, you can use the same health check configuration in all your gRPC
applications. This approach requires you to:
1. Find the gRPC "health" module in your favorite language and start using it
(example [Go library](https://godoc.org/github.com/grpc/grpc-go/health)).
2. Ship the
[grpc_health_probe](https://github.com/grpc-ecosystem/grpc-health-probe/)
binary in your container.
3. [Configure](https://github.com/grpc-ecosystem/grpc-health-probe/tree/1329d682b4232c102600b5e7886df8ffdcaf9e26#example-grpc-health-checking-on-kubernetes)
Kubernetes "exec" probe to invoke the "grpc_health_probe" tool in the
container.
In this case, executing "grpc_health_probe" will call your gRPC server over
`localhost`, since they are in the same pod.
## What's next
**grpc-health-probe** project is still in its early days and it needs your
feedback. It supports a variety of features like communicating with TLS servers
and configurable connection/RPC timeouts.
If you are running a gRPC server on Kubernetes today, try using the gRPC Health
Protocol and try the grpc-health-probe in your deployments, and [give
feedback](https://github.com/grpc-ecosystem/grpc-health-probe/).
## Further reading
- Protocol: [GRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/v1.15.0/doc/health-checking.md) ([health.proto](https://github.com/grpc/grpc/blob/v1.15.0/src/proto/grpc/health/v1/health.proto))
- Documentation: [Kubernetes liveness and readiness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
- Article: [Advanced Kubernetes Health Check Patterns](https://ahmet.im/blog/advanced-kubernetes-health-checks/)

Binary file not shown.

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 149 KiB