102 lines
4.9 KiB
Markdown
102 lines
4.9 KiB
Markdown
---
|
|
layout: blog
|
|
title: 'Health checking gRPC servers on Kubernetes'
|
|
date: 2018-10-01
|
|
---
|
|
|
|
**Author**: [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google)
|
|
|
|
**Update (December 2021):** _Kubernetes now has built-in gRPC health probes starting in v1.23.
|
|
To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe).
|
|
This article was originally written about an external tool to achieve the same task._
|
|
|
|
[gRPC](https://grpc.io) is on its way to becoming the lingua franca for
|
|
communication between cloud-native microservices. If you are deploying gRPC
|
|
applications to Kubernetes today, you may be wondering about the best way to
|
|
configure health checks. In this article, we will talk about
|
|
[grpc-health-probe](https://github.com/grpc-ecosystem/grpc-health-probe/), a
|
|
Kubernetes-native way to health check gRPC apps.
|
|
|
|
If you're unfamiliar, Kubernetes [health
|
|
checks](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
|
|
(liveness and readiness probes) is what's keeping your applications available
|
|
while you're sleeping. They detect unresponsive pods, mark them unhealthy, and
|
|
cause these pods to be restarted or rescheduled.
|
|
|
|
Kubernetes [does not
|
|
support](https://github.com/kubernetes/kubernetes/issues/21493) gRPC health
|
|
checks natively. This leaves the gRPC developers with the following three
|
|
approaches when they deploy to Kubernetes:
|
|
|
|
[](/images/blog/2019-09-30-health-checking-grpc/options.png)
|
|
|
|
|
|
1. **httpGet probe:** Cannot be natively used with gRPC. You need to refactor
|
|
your app to serve both gRPC and HTTP/1.1 protocols (on different port
|
|
numbers).
|
|
2. **tcpSocket probe:** Opening a socket to gRPC server is not meaningful,
|
|
since it cannot read the response body.
|
|
3. **exec probe:** This invokes a program in a container's ecosystem
|
|
periodically. In the case of gRPC, this means you implement a health RPC
|
|
yourself, then write and ship a client tool with your container.
|
|
|
|
Can we do better? Absolutely.
|
|
|
|
## Introducing “grpc-health-probe”
|
|
|
|
To standardize the "exec probe" approach mentioned above, we need:
|
|
|
|
- a **standard** health check "protocol" that can be implemented in any gRPC
|
|
server easily.
|
|
- a **standard** health check "tool" that can query the health protocol easily.
|
|
|
|
Thankfully, gRPC has a [standard health checking
|
|
protocol](https://github.com/grpc/grpc/blob/v1.15.0/doc/health-checking.md). It
|
|
can be used easily from any language. Generated code and the utilities for
|
|
setting the health status are shipped in nearly all language implementations of
|
|
gRPC.
|
|
|
|
If you
|
|
[implement](https://github.com/grpc/grpc/blob/v1.15.0/src/proto/grpc/health/v1/health.proto)
|
|
this health check protocol in your gRPC apps, you can then use a standard/common
|
|
tool to invoke this `Check()` method to determine server status.
|
|
|
|
The next thing you need is the "standard tool", and it's the
|
|
[**grpc-health-probe**](https://github.com/grpc-ecosystem/grpc-health-probe/).
|
|
|
|
<a href='/images/blog/2019-09-30-health-checking-grpc/grpc_health_probe.png'>
|
|
<img width="768" title='grpc-health-probe on kubernetes'
|
|
src='/images/blog/2019-09-30-health-checking-grpc/grpc_health_probe.png'/>
|
|
</a>
|
|
|
|
With this tool, you can use the same health check configuration in all your gRPC
|
|
applications. This approach requires you to:
|
|
|
|
1. Find the gRPC "health" module in your favorite language and start using it
|
|
(example [Go library](https://godoc.org/github.com/grpc/grpc-go/health)).
|
|
2. Ship the
|
|
[grpc_health_probe](https://github.com/grpc-ecosystem/grpc-health-probe/)
|
|
binary in your container.
|
|
3. [Configure](https://github.com/grpc-ecosystem/grpc-health-probe/tree/1329d682b4232c102600b5e7886df8ffdcaf9e26#example-grpc-health-checking-on-kubernetes)
|
|
Kubernetes "exec" probe to invoke the "grpc_health_probe" tool in the
|
|
container.
|
|
|
|
In this case, executing "grpc_health_probe" will call your gRPC server over
|
|
`localhost`, since they are in the same pod.
|
|
|
|
## What's next
|
|
|
|
**grpc-health-probe** project is still in its early days and it needs your
|
|
feedback. It supports a variety of features like communicating with TLS servers
|
|
and configurable connection/RPC timeouts.
|
|
|
|
If you are running a gRPC server on Kubernetes today, try using the gRPC Health
|
|
Protocol and try the grpc-health-probe in your deployments, and [give
|
|
feedback](https://github.com/grpc-ecosystem/grpc-health-probe/).
|
|
|
|
## Further reading
|
|
|
|
- Protocol: [GRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/v1.15.0/doc/health-checking.md) ([health.proto](https://github.com/grpc/grpc/blob/v1.15.0/src/proto/grpc/health/v1/health.proto))
|
|
- Documentation: [Kubernetes liveness and readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
|
|
- Article: [Advanced Kubernetes Health Check Patterns](https://ahmet.im/blog/advanced-kubernetes-health-checks/)
|