79 lines
3.5 KiB
Markdown
79 lines
3.5 KiB
Markdown
---
|
|
layout: blog
|
|
title: "Kubernetes 1.27: Query Node Logs Using The Kubelet API"
|
|
date: 2023-04-21
|
|
slug: node-log-query-alpha
|
|
author: >
|
|
Aravindh Puthiyaparambil (Red Hat)
|
|
---
|
|
|
|
Kubernetes 1.27 introduced a new feature called _Node log query_ that allows
|
|
viewing logs of services running on the node.
|
|
|
|
## What problem does it solve?
|
|
Cluster administrators face issues when debugging malfunctioning services
|
|
running on the node. They usually have to SSH or RDP into the node to view the
|
|
logs of the service to debug the issue. The _Node log query_ feature helps with
|
|
this scenario by allowing the cluster administrator to view the logs using
|
|
_kubectl_. This is especially useful with Windows nodes where you run into the
|
|
issue of the node going to the ready state but containers not coming up due to
|
|
CNI misconfigurations and other issues that are not easily identifiable by
|
|
looking at the Pod status.
|
|
|
|
## How does it work?
|
|
|
|
The kubelet already has a _/var/log/_ viewer that is accessible via the node
|
|
proxy endpoint. The feature supplements this endpoint with a shim that shells
|
|
out to `journalctl`, on Linux nodes, and the `Get-WinEvent` cmdlet on Windows
|
|
nodes. It then uses the existing filters provided by the commands to allow
|
|
filtering the logs. The kubelet also uses heuristics to retrieve the logs.
|
|
If the user is not aware if a given system services logs to a file or to the
|
|
native system logger, the heuristics first checks the native operating system
|
|
logger and if that is not available it attempts to retrieve the first logs
|
|
from `/var/log/<servicename>` or `/var/log/<servicename>.log` or
|
|
`/var/log/<servicename>/<servicename>.log`.
|
|
|
|
On Linux we assume that service logs are available via journald, and that
|
|
`journalctl` is installed. On Windows we assume that service logs are available
|
|
in the application log provider. Also note that fetching node logs is only
|
|
available if you are authorized to do so (in RBAC, that's **get** and
|
|
**create** access to `nodes/proxy`). The privileges that you need to fetch node
|
|
logs also allow elevation-of-privilege attacks, so be careful about how you
|
|
manage them.
|
|
|
|
## How do I use it?
|
|
|
|
To use the feature, ensure that the `NodeLogQuery`
|
|
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is
|
|
enabled for that node, and that the kubelet configuration options
|
|
`enableSystemLogHandler` and `enableSystemLogQuery` are both set to true. You can
|
|
then query the logs from all your nodes or just a subset. Here is an example to
|
|
retrieve the kubelet service logs from a node:
|
|
```shell
|
|
# Fetch kubelet logs from a node named node-1.example
|
|
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
|
|
```
|
|
|
|
You can further filter the query to narrow down the results:
|
|
```shell
|
|
# Fetch kubelet logs from a node named node-1.example that have the word "error"
|
|
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"
|
|
```
|
|
|
|
You can also fetch files from `/var/log/` on a Linux node:
|
|
```shell
|
|
kubectl get --raw "/api/v1/nodes/<insert-node-name-here>/proxy/logs/?query=/<insert-log-file-name-here>"
|
|
```
|
|
|
|
You can read the
|
|
[documentation](/docs/concepts/cluster-administration/system-logs/#log-query)
|
|
for all the available options.
|
|
|
|
## How do I help?
|
|
|
|
Please use the feature and provide feedback by opening GitHub issues or
|
|
reaching out to us on the
|
|
[#sig-windows](https://kubernetes.slack.com/archives/C0SJ4AFB7) channel on the
|
|
Kubernetes Slack or the SIG Windows
|
|
[mailing list](https://groups.google.com/g/kubernetes-sig-windows).
|