What Is Kubernetes Logging?

On a traditional VM, a crashed application leaves its log file sitting on disk for you to inspect. In Kubernetes, a dead pod takes its logs with it. They can disappear before you've even finished reading the stack trace.

Kubernetes logging is the chain of mechanisms that takes whatever your container writes to stdout and stderr, hands it to the container runtime, and stores it as a file on the node where the kubelet can serve it back to you through kubectl logsCopy. That's the whole base layer, and it's more fragile than most people expect. Those log files are tied to the lifecycle of the container, so when a pod is deleted or rescheduled, its logs go with it.

Everything else in the Kubernetes logging story is built to work around that fact.

How a log line travels through Kubernetes

The default path is simpler than the diagrams make it look. Your application writes a line to stdout or stderr. The container runtime (containerd or CRI-O on modern clusters) captures both streams and writes them to a file on the node in a structured format, one line per entry, tagged with a timestamp and the stream name.

You can see exactly what that looks like on the node:

12
2026-06-15T10:30:45.123456789Z stdout F Request processed in 45ms
2026-06-15T10:30:45.234567890Z stderr F Connection to upstream failed

Each line carries an RFC 3339 timestamp, the stream it came from (stdoutCopy or stderrCopy), a tag (FCopy for a full line, PCopy for a partial one that got split), and then the actual message. These files live under /var/log/pods/Copy on the node, with symlinks in /var/log/containers/Copy pointing back to them. When you run kubectl logsCopy, the kubelet on that node reads directly from these files and streams the contents back to you. There's no central log store involved — you're reading a file off one specific machine.

This design is why the best practice is to write logs to stdout and stderr rather than to a file inside the container. If your application writes to its own log file instead, the kubelet has no idea that file exists, and kubectl logsCopy will return nothing. The container's stdout is the contract Kubernetes understands.

Where the logs come from

There are three distinct sources of logs in a cluster, and conflating them is a common source of confusion.

Container (application) logs are everything your own workloads emit. This is the layer you interact with most through kubectl logsCopy, and the one the rest of this article focuses on.

Node-level logs belong to the components running on each machine. The kubelet itself is the notable exception to the stdout pattern: it runs as a native systemd service rather than a container, so it writes to journald, and you read it with journalctl -u kubeletCopy rather than kubectl logsCopy.

Control plane logs come from the API server, scheduler, controller manager, and etcd. On most managed clusters these components run as containers, so their logs flow through the same stdout-to-node-file path as your applications. On a self-managed cluster you may instead find them under /var/log/Copy on the control plane nodes. These logs matter more than people assume: a performance or security problem in the control plane affects the entire cluster, not just one workload.

The log rotation trap

The kubelet rotates container log files to keep them from filling the node's disk. Two settings in the kubelet configuration control this, and their defaults catch teams off guard under load:

12345
# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 10Mi   # rotate when a file hits this size
containerLogMaxFiles: 5     # keep at most this many files per container

With the defaults of 10MiCopy and 5Copy, a single container keeps at most roughly 50 MiB of logs on the node before the oldest rotated file is deleted. A chatty service under heavy traffic can blow through that in seconds, which means by the time you go looking, the log lines you needed have already been rotated out and discarded. This is not a bug; it's the kubelet doing exactly what it was told. If you're seeing gaps in logs for a high-volume pod, raising containerLogMaxSizeCopy is the first lever to pull — though the real backstop against logs filling a node is the kubelet's eviction manager, which marks the node as under disk pressure and begins evicting pods once free space on the node filesystem drops below the nodefs.availableCopy threshold (10% by default).

One detail worth internalizing: each rotation belongs to a single container instance. If a container crashes and restarts repeatedly, or the pod gets evicted, the kubelet retains the current log and the one from just before the last restart, but everything older is gone. You cannot rely on the kubelet as a durable log store.

Why you need cluster-level logging

Put the ephemerality and the rotation together and the conclusion is unavoidable. The node-local logging that Kubernetes gives you out of the box is fine for live debugging and useless for anything that has to survive a pod restart, a node failure, or an incident postmortem written the next morning.

The standard solution is cluster-level logging, which decouples log storage from the lifecycle of pods, nodes, and containers. Kubernetes deliberately provides no built-in implementation; it gives you the architecture and expects you to bring the tooling. The dominant pattern is a node-level logging agent deployed as a DaemonSet, so exactly one agent runs on every node:

12345678910111213141516171819202122232425
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
  namespace: logging
spec:
  selector:
    matchLabels:
      app: log-agent
  template:
    metadata:
      labels:
        app: log-agent
    spec:
      containers:
        - name: agent
          image: fluent/fluent-bit:5.0.7
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

The agent mounts the node's /var/logCopy directory, tails the same container log files the kubelet writes, enriches each line with Kubernetes metadata such as pod name, namespace, and labels, and ships it to a backend that stores and indexes it independently of the cluster. Because the agent reads the files the kubelet already produces, you keep the benefits of the stdout pattern and add durability on top. Fluent Bit and Fluentd are the most common choices for the agent role; both are CNCF projects. Dash0 has native integrations for both if you're already running either in your cluster.

Common pitfalls

The mistake that wastes the most time is assuming logs are durable. You're debugging an intermittent failure, the relevant pod has restarted, you run kubectl logsCopy, and you get the logs of the new healthy container instead of the dead one. The fix is kubectl logs <pod> --previousCopy, which returns the logs from the prior container instance — but only if it hasn't already been rotated away. The real fix is shipping logs off the node before you need them.

A subtler trap is the sidecar logging agent pattern. Instead of one DaemonSet agent per node, some teams run a logging agent as a sidecar container in every pod. It works, and it's occasionally necessary for pods that can't write to stdout. But it multiplies resource consumption by the number of pods rather than the number of nodes. On a busy cluster that's a substantial and often invisible cost. Prefer streaming to stdout and letting a single node-level agent do the collection unless you have a specific reason not to.

Teams also forget that multi-line logs get split. A Java stack trace written as twenty separate lines arrives as twenty separate log entries unless your collection agent is configured to reassemble them. That PCopy tag in the CRI log format exists precisely because the runtime splits lines longer than its buffer. Your agent needs to know how to stitch them back together, or your stack traces will be scattered across entries.

Final thoughts

Node-local logs in Kubernetes are ephemeral by design. The logs you most need during an incident are exactly the ones most likely to have already been rotated away or deleted with a dead pod, so the only approach that holds up is shipping them somewhere durable before the incident happens.

The Dash0 Kubernetes Operator handles the collection layer for you — it installs an OpenTelemetry Collector into your cluster and automatically collects logs from your workloads, with no DaemonSet to configure manually. Those logs are stored alongside metrics and distributed traces, so when a pod dies you still have its logs, correlated with the traces and Kubernetes cluster metrics that show what was happening around it. Start a free trial to see your cluster's logs, metrics, and traces in one view. No credit card required.