Document structured logging
* Separate system logs from Logging Architecture as it mainly covers application logging * Rename "Metics For The Kubernetes Control Plane" to "Metrics For The Kubernetes System Components" as it covers non-control plane components * Rename files to `system-logs.md` and `system-metrics.md` * Remove trailing whitespaces * Add sections about Klog and Structured Logging Co-authored-by: Celeste Horgan <celeste@cncf.io> Co-authored-by: Zach Corleissen <zacharysarah@users.noreply.github.com> Co-authored-by: Tim Bannister <tim@scalefactory.com> Co-authored-by: Paweł Kępka <pkepka@google.com>pull/21202/head
Marek Siarkowicz
Marek Siarkowicz
parent
d6b7bf5f38
commit
f9833ee09a
|
|
@ -9,13 +9,10 @@ weight: 60
|
|||
|
||||
<!-- overview -->
|
||||
|
||||
Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism; as such, most container engines are likewise designed to support some kind of logging. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams.
|
||||
Application logs can help you understand what is happening inside your application. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism; as such, most container engines are likewise designed to support some kind of logging. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams.
|
||||
|
||||
However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution. For example, if a container crashes, a pod is evicted, or a node dies, you'll usually still want to access your application's logs. As such, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called _cluster-level-logging_. Cluster-level logging requires a separate backend to store, analyze, and query logs. Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster.
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- body -->
|
||||
|
||||
Cluster-level logging architectures are described in assumption that
|
||||
|
|
@ -98,28 +95,6 @@ the rotation and there are two files, one 10MB in size and one empty,
|
|||
|
||||
[cosConfigureHelper]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh
|
||||
|
||||
### System component logs
|
||||
|
||||
There are two types of system components: those that run in a container and those
|
||||
that do not run in a container. For example:
|
||||
|
||||
* The Kubernetes scheduler and kube-proxy run in a container.
|
||||
* The kubelet and container runtime, for example Docker, do not run in containers.
|
||||
|
||||
On machines with systemd, the kubelet and container runtime write to journald. If
|
||||
systemd is not present, they write to `.log` files in the `/var/log` directory.
|
||||
System components inside containers always write to the `/var/log` directory,
|
||||
bypassing the default logging mechanism. They use the [klog][klog]
|
||||
logging library. You can find the conventions for logging severity for those
|
||||
components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md).
|
||||
|
||||
Similarly to the container logs, system component logs in the `/var/log`
|
||||
directory should be rotated. In Kubernetes clusters brought up by
|
||||
the `kube-up.sh` script, those logs are configured to be rotated by
|
||||
the `logrotate` tool daily or once the size exceeds 100MB.
|
||||
|
||||
[klog]: https://github.com/kubernetes/klog
|
||||
|
||||
## Cluster-level logging architectures
|
||||
|
||||
While Kubernetes does not provide a native solution for cluster-level logging, there are several common approaches you can consider. Here are some options:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,119 @@
|
|||
---
|
||||
reviewers:
|
||||
- dims
|
||||
- 44past4
|
||||
title: System Logs
|
||||
content_type: concept
|
||||
weight: 60
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
System component logs record events happening in cluster, which can be very useful for debugging.
|
||||
You can configure log verbosity to see more or less detail.
|
||||
Logs can be as coarse-grained as showing errors within a component, or as fine-grained as showing step-by-step traces of events (like HTTP access logs, pod state changes, controller actions, or scheduler decisions).
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Klog
|
||||
|
||||
klog is the Kubernetes logging library. [klog](https://github.com/kubernetes/klog)
|
||||
generates log messages for the Kubernetes system components.
|
||||
|
||||
For more information about klog configuration, see the [Command line tool reference](/docs/reference/command-line-tools-reference/).
|
||||
|
||||
An example of the klog native format:
|
||||
```
|
||||
I1025 00:15:15.525108 1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756]
|
||||
```
|
||||
|
||||
### Structured Logging
|
||||
|
||||
{{< feature-state for_k8s_version="v1.19" state="alpha" >}}
|
||||
|
||||
{{<warning>}}
|
||||
Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages.
|
||||
|
||||
Log formatting and value serialization are subject to change.
|
||||
{{< /warning>}}
|
||||
|
||||
Structured logging is a effort to introduce a uniform structure in log messages allowing for easy extraction of information, making logs easier and cheaper to store and process.
|
||||
New message format is backward compatible and enabled by default.
|
||||
|
||||
Format of structured logs:
|
||||
```
|
||||
<klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ...
|
||||
```
|
||||
|
||||
Example:
|
||||
```
|
||||
I1025 00:15:15.525108 1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready"
|
||||
```
|
||||
|
||||
|
||||
### JSON log format
|
||||
|
||||
{{< feature-state for_k8s_version="v1.19" state="alpha" >}}
|
||||
|
||||
{{<warning >}}
|
||||
JSON output does not support many standard klog flags. For list of unsupported klog flags, see the [Command line tool reference](/docs/reference/command-line-tools-reference/).
|
||||
|
||||
Not all logs are guaranteed to be written in JSON format (for example, during process start). If you intend to parse logs, make sure you can handle log lines that are not JSON as well.
|
||||
|
||||
Field names and JSON serialization are subject to change.
|
||||
{{< /warning >}}
|
||||
|
||||
The `--logging-format=json` flag changes the format of logs from klog native format to JSON format.
|
||||
Example of JSON log format (pretty printed):
|
||||
```json
|
||||
{
|
||||
"ts": 1580306777.04728,
|
||||
"v": 4,
|
||||
"msg": "Pod status updated",
|
||||
"pod":{
|
||||
"name": "nginx-1",
|
||||
"namespace": "default"
|
||||
},
|
||||
"status": "ready"
|
||||
}
|
||||
```
|
||||
|
||||
Keys with special meaning:
|
||||
* `ts` - timestamp as Unix time (required, float)
|
||||
* `v` - verbosity (required, int, default 0)
|
||||
* `err` - error string (optional, string)
|
||||
* `msg` - message (required, string)
|
||||
|
||||
|
||||
List of components currently supporting JSON format:
|
||||
* {{< glossary_tooltip term_id="kube-controller-manager" text="kube-controller-manager" >}}
|
||||
* {{< glossary_tooltip term_id="kube-apiserver" text="kube-apiserver" >}}
|
||||
* {{< glossary_tooltip term_id="kube-scheduler" text="kube-scheduler" >}}
|
||||
* {{< glossary_tooltip term_id="kubelet" text="kubelet" >}}
|
||||
|
||||
### Log verbosity level
|
||||
|
||||
The `-v` flag controls log verbosity. Increasing the value increases the number of logged events. Decreasing the value decreases the number of logged events.
|
||||
Increasing verbosity settings logs increasingly less severe events. A verbosity setting of 0 logs only critical events.
|
||||
|
||||
### Log location
|
||||
|
||||
There are two types of system components: those that run in a container and those
|
||||
that do not run in a container. For example:
|
||||
|
||||
* The Kubernetes scheduler and kube-proxy run in a container.
|
||||
* The kubelet and container runtime, for example Docker, do not run in containers.
|
||||
|
||||
On machines with systemd, the kubelet and container runtime write to journald.
|
||||
Otherwise, they write to `.log` files in the `/var/log` directory.
|
||||
System components inside containers always write to `.log` files in the `/var/log` directory,
|
||||
bypassing the default logging mechanism.
|
||||
Similar to the container logs, you should rotate system component logs in the `/var/log` directory.
|
||||
In Kubernetes clusters created by the `kube-up.sh` script, log rotation is configured by the `logrotate` tool.
|
||||
The `logrotate` tool rotates logs daily, or once the log size is greater than 100MB.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Read about the [Kubernetes Logging Architecture](/docs/concepts/cluster-administration/logging/)
|
||||
* Read about [Structured Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging)
|
||||
* Read about the [Conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md)
|
||||
|
|
@ -1,22 +1,19 @@
|
|||
---
|
||||
title: Metrics For The Kubernetes Control Plane
|
||||
title: Metrics For Kubernetes System Components
|
||||
reviewers:
|
||||
- brancz
|
||||
- logicalhan
|
||||
- RainbowMango
|
||||
content_type: concept
|
||||
weight: 60
|
||||
aliases:
|
||||
- controller-metrics.md
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
System component metrics can give a better look into what is happening inside them. Metrics are particularly useful for building dashboards and alerts.
|
||||
|
||||
Metrics in Kubernetes control plane are emitted in [prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/) and are human readable.
|
||||
|
||||
|
||||
Kubernetes components emit metrics in [Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/).
|
||||
This format is structured plain text, designed so that people and machines can both read it.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
|
|
@ -39,7 +36,7 @@ Note that {{< glossary_tooltip term_id="kubelet" text="kubelet" >}} also exposes
|
|||
|
||||
If your cluster uses {{< glossary_tooltip term_id="rbac" text="RBAC" >}}, reading metrics requires authorization via a user, group or ServiceAccount with a ClusterRole that allows accessing `/metrics`.
|
||||
For example:
|
||||
```
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
|
|
@ -124,8 +121,6 @@ cloudprovider_gce_api_request_duration_seconds { request = "detach_disk"}
|
|||
cloudprovider_gce_api_request_duration_seconds { request = "list_disk"}
|
||||
```
|
||||
|
||||
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Read about the [Prometheus text format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) for metrics
|
||||
|
|
@ -89,6 +89,8 @@
|
|||
/docs/concepts/cluster-administration/network-plugins/ /docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/ 301
|
||||
/docs/concepts/cluster-administration/out-of-resource/ /docs/tasks/administer-cluster/out-of-resource/ 301
|
||||
/docs/concepts/cluster-administration/resource-usage-monitoring /docs/tasks/debug-application-cluster/resource-usage-monitoring/ 301
|
||||
/docs/concepts/cluster-administration/monitoring/ /docs/concepts/cluster-administration/system-metrics/ 301
|
||||
/docs/concepts/cluster-administration/controller-metrics/ /docs/concepts/cluster-administration/system-metrics/ 301
|
||||
/docs/concepts/cluster-administration/sysctl-cluster/ /docs/tasks/administer-cluster/sysctl-cluster/ 301
|
||||
/docs/concepts/cluster-administration/static-pod/ /docs/tasks/administer-cluster/static-pod/ 301
|
||||
/docs/concepts/clusters/logging/ /docs/concepts/cluster-administration/logging/ 301
|
||||
|
|
|
|||
Loading…
Reference in New Issue