Linux VMs root cause analysis guides

Kubernetes gives you crash loops. VMs give you slow leaks. The difference matters, because it changes how you investigate.

On a Kubernetes pod, when something breaks, it usually breaks in a way that repeats, an OOMKill, a CrashLoopBackOff, a failing readiness probe. The scheduler brings it back and it fails again, and the pattern is often visible on the first alert. On a long-lived VM, there is no scheduler and no reset. A process that leaks a little each day just accumulates until it hits a wall. A partition that fills at a steady slope just keeps filling until a write returns ENOSPC. A table that grows past the size where a plan works keeps growing until one query times out. The failure does not repeat. It saturates.

That single difference is why VM incidents are hard, and why the usual instinct of looking at the most recent deploy so often points at the wrong week.

Why do Linux VM incidents feel harder to debug than Kubernetes ones?#

Because the failure has been building longer than the investigation naturally reaches back.

Most incident-response instincts come from environments that fail fast and reset often. In those environments, the change responsible is usually something that shipped in the last few hours. On a VM, the shape of failure is different. A leak accumulates for days. A partition fills at a rate you cannot see in a five-minute window. A configuration management refactor silently drops a logrotate rule, and nothing happens for two weeks, until the disk fills. When the page finally fires, correlating with the last deploy points at code that had nothing to do with the actual cause.

So the first move in any slow-onset VM incident is different from the fast-fail one. Look at the shape of the resource curve before looking at the resource. A monotonic slope means either a leak or a missing reclaim mechanism. A bursty curve tracking traffic is a capacity question. And crucially, extrapolating that slope backward to its inflection point often gives you a window in change history that is far more precise than anything in the logs.

The Linux VM troubleshooting guides#

Each guide below is built on a real investigation and covers a specific slow-failure pattern.

Disk full from silent config drift. A disk fills not because traffic grew, but because a configuration management change silently stopped a rotation or cleanup rule from being applied. In one investigation, an Ansible refactor accidentally dropped a logrotate config, and 14 days of unrotated logs at 87 GB filled the 100 GB /var volume, triggering ENOSPC errors and HTTP 500s. The cause was two weeks old and lived in the configuration management repo, not the application repo. Covers the du-vs-df diagnostic, when to reach for lsof +L1, and why the configuration management repository is the first place to look on any slow disk incident.

JVM memory leaks and Linux OOM kills on VMs. A JVM process on a VM slowly accumulates heap over hours until the Linux OOM killer terminates it, killing the process from underneath the JVM without a Java-side exception. In one investigation, a Hibernate session cache leak in a nightly batch job grew a payment-gateway JVM from 4 GB to 14 GB over six hours before the kernel killed it, causing eight minutes of downtime. Covers the crucial distinction between the kernel OOM killer (which writes its victim, RSS, and cgroup to the kernel log) and a JVM throwing OutOfMemoryError with heap intact, because those are different failures and confusing them sends the investigation into the wrong process.

A third failure mode common on VMs, connection pool exhaustion at peak traffic when a schema change silently made a query slower, is covered in the Databases connection pool exhaustion guide. The root cause is database behavior rather than a VM-specific problem, so the Databases guide is the canonical home. Both this hub and the Databases hub cross-link because the symptom often surfaces first on a VM cluster.

How do you triage a slow-onset Linux VM incident?#

Three moves, applied in order.

Look at the shape of the resource curve before you look at the resource. Monotonic growth with a near-constant slope means a leak or a missing reclaim mechanism, and the slope itself is diagnostic: extrapolating back to the inflection point gives you a window to search in change history, which is frequently more precise than anything in the logs. Bursty growth that tracks traffic is a capacity question instead. Getting this distinction right in the first five minutes decides whether you are debugging code or debugging a rate limit.

For memory, establish who killed what. The Linux kernel OOM killer writes its victim, RSS, and the triggering cgroup to the kernel log, which is a different failure from a JVM throwing OutOfMemoryError with heap intact. Confusing the two sends the investigation into the wrong process. On JVM workloads especially, always confirm which layer did the killing before touching heap settings.

For disk, compare du against df. Disagreement means deleted files still held open by a running process, the classic signature of a rotation that stopped or a truncate that never happened. lsof +L1 names the holder. Agreement means real growth, so find the directory and the writer. Either way, check the configuration management repository rather than only the application repository. A VM is the output of its last converge, and a refactor that reorganized templates can remove a logrotate rule or a sysctl without any visible change to the service it silently protected.

You can see these worked end to end, including the paths that were ruled out, on the Linux VMs examples page. Several of these patterns cross into other layers of the stack: a connection-pool exhaustion often begins in the Databases layer, and an OOM comparison bridges to Kubernetes container-local kills, which behave differently from a kernel OOM on a VM. For the underlying mechanics, the Linux kernel documentation and the systemd journal documentation are the primary references, along with the JVM engine documentation for whichever runtime is in play.

Frequently asked questions#

Why do VM incidents often trace back to a change weeks earlier? Because on long-lived VMs, failures accumulate rather than repeat. A leak that adds a small amount of memory each day only trips a page when it saturates, and by then the change responsible is not in the recent deploy window.

What is the difference between a Linux OOM kill and a JVM OutOfMemoryError? The kernel OOM killer terminates a process from outside, writing its victim to the kernel log, when the system runs out of memory. A JVM OutOfMemoryError is thrown from inside the JVM when its own heap is exhausted while the system may still have memory available. They are different failures and require different fixes.

How do I know if my disk is full from real growth or from held-open deleted files? Compare du against df. If they agree, growth is real. If df shows the volume as fuller than du, a process is holding deleted files open. lsof +L1 names the process.

What is the fastest way to spot a memory leak on a VM? Look at the slope of the memory curve over hours or days. A monotonic near-constant slope is a leak. A curve that tracks traffic is a capacity question, not a leak.

Why should I check the configuration management repository during a VM incident? Because a VM is the output of its last converge. Configuration management refactors can silently remove a logrotate rule, a sysctl, or a systemd unit that was quietly protecting the service. If the alert fires two weeks after such a change, no application deploy will look suspicious.