On a long-lived VM there is no scheduler to restart the process and no event stream that names what happened, so failures accumulate instead of repeating. Heap grows a little each day for a week, a partition fills at a steady slope, a table crosses the size where a sequential scan stops being survivable. By the time the page fires, the change responsible is often weeks old: a configuration management refactor that silently stopped shipping a file, a migration that added rows faster than the plan could absorb. That gap defeats the usual first move of correlating with the most recent deploy. The three investigations here are all slow failures, and in each one the fix was elsewhere in time from the alert.
Ansible playbook refactor accidentally removed logrotate config. 14 days of unrotated logs (87 GB) filled the 100 GB /var volume, causing ENOSPC errors and HTTP 500s.
Missing database index after schema migration caused 42-second queries (from 50ms). Connection pool across 3 app servers exhausted during morning peak, returning HTTP 503 for all users.
Hibernate session cache leak in batch reconciliation job grew JVM heap from 4 GB to 14 GB over 6 hours. Linux OOM killer terminated the process, causing 8 minutes of downtime.
Look at the shape of the resource curve before looking at the resource. Monotonic growth with a near-constant slope means either a leak or a missing reclaim mechanism, and the slope is useful: extrapolating back to the inflection point gives a window to search in change history, which is frequently more precise than anything in the logs. Bursty growth that tracks traffic is a capacity question instead. For memory, establish who killed what. The kernel OOM killer writes its victim, RSS, and the triggering cgroup to the kernel log, which is a different failure from a JVM throwing OutOfMemoryError with heap intact, and confusing the two sends the investigation into the wrong process. For disk, compare du against df. Disagreement means deleted files still held open by a running process, the signature of a rotation that stopped or a truncate that never happened, and lsof +L1 names the holder. Agreement means real growth, so find the directory and the writer. In all three cases, check the configuration management repository rather than only the application repository. A VM is the output of its last converge, and a refactor that reorganized templates can remove a logrotate rule or a sysctl without any change to the service it silently protected.