How to fix a JVM killed by the Linux OOM killer on a VM (heap leak, no OutOfMemoryError)?

The process is gone, systemd restarted it, and there is nothing in the application log. That silence is the diagnosis: nothing inside the JVM got a chance to report anything.

What is the difference between a kernel OOM kill and a JVM OutOfMemoryError?#

One is the operating system terminating your process from outside. The other is your process reporting its own exhaustion from inside. They look similar on a dashboard and share almost nothing else.

The kernel OOM killer fires when the host runs out of memory. It selects a victim by score, sends SIGKILL, and writes a record to the kernel ring buffer. The process gets no signal handler, no shutdown hook, no finally block and no chance to log. From inside the application, the event is invisible.

A JVM OutOfMemoryError is thrown when the JVM's own heap is exhausted while the host may still have plenty of memory free. It unwinds as a Java Error, it appears in the application log with a stack trace, and shutdown hooks generally run.

                        kernel OOM kill              JVM OutOfMemoryErrorwho decides             the kernel                   the JVMconstraint hit          host or cgroup memory        -Xmx heap ceilingevidence lands in       dmesg / journal              application logJava stack trace        none                         yesshutdown hooks run      no                           usuallyheap dump on exit       no                           with -XX:+HeapDumpOnOutOfMemoryError 

The practical consequence is what to do first. An empty application log around the time of death is not missing evidence, it is evidence. It means the killing happened at a layer the JVM cannot observe, so stop grepping application logs and go read the kernel's.

How do you confirm the kernel did the killing?#

Read the kernel log. It names the victim explicitly, and the record is unambiguous.

# -T renders human timestamps, which is what you need to line the# kill up against a metrics graph or a cron schedule.sudo dmesg -T | grep -i -A3 'out of memory' # On a systemd host the journal keeps this across reboots, which# dmesg does not. Widen the window if the restart already happened.sudo journalctl -k --since '-6h' | grep -i 'killed process\|oom' 

[2026-03-28T03:12:04+0530] Out of memory: Killed process 2847 (java)  total-vm:18874368kB, anon-rss:14893056kB, file-rss:12288kB  oom_score_adj: 0 

Three fields carry the information. The process name and PID identify the victim, and it is worth confirming it is the process you assumed: the OOM killer picks by score, so the biggest consumer is the usual victim but not a guaranteed one. anon-rss is the anonymous resident set, the memory actually held, and comparing it against host RAM tells you whether this process alone accounts for the exhaustion or whether the host was oversubscribed by several. oom_score_adj shows whether anything had tuned the victim's priority.

Here anon-rss of roughly 14.2 GB on a host with about 15 GiB usable is the whole answer to "was this one process or the whole box". One process. That rules out a noisy neighbour and hardware fault in a single line.

Check the peers next, because a workload-shaped cause and a host-shaped cause separate immediately across hosts running the same code.

# Same service, same traffic, different hosts. If only one is in# trouble, the differentiator is what that host does differently.for h in prod-vm-01 prod-vm-02 prod-vm-03; do  printf '%-12s ' "$h"  ssh "$h" "free -m | awk '/Mem:/ {printf \"%.0f%%\n\", \$3/\$2*100}'"done 

Why does raising -Xmx not fix it?#

Because a leak has no ceiling to be sized against. A larger heap changes when the process dies, not whether.

The distinction is the shape of the curve. Memory that climbs and then plateaus is a working set, and if the plateau sits above -Xmx then the heap is genuinely undersized. Memory that climbs monotonically and never levels off is retention, and no value of -Xmx is large enough because the requirement grows with the work processed rather than with the concurrent work in flight.

time   heap     note21:00  4.1 GB   baseline (batch starts)23:00  6.3 GB   climbing01:00  8.9 GB   past -Xmx8g (off-heap and native)03:00  13.8 GB  near host limit03:12  14.2 GB  OOM killed 

Two things are worth reading off that table. There is no plateau anywhere, so this is retention rather than sizing. And the process was consuming more than its configured -Xmx8g well before it died, which is the reminder that -Xmx bounds the Java heap and not the process. Metaspace, thread stacks, code cache, direct byte buffers and the allocator's own overhead all sit outside it, which is why a JVM configured for 8 GB can present 14 GB of RSS to the kernel.

That gap is also why the kernel got involved at all. Had the growth been confined to the Java heap, the JVM would have thrown OutOfMemoryError at its own ceiling and told you so.

How do you tell a leak from ordinary GC pressure?#

Look at what major collections actually reclaim. A collector working hard and freeing nothing is the signature.

Under ordinary pressure, collections are frequent and expensive but old generation occupancy drops noticeably each time. Under a leak, the collector runs, pays the full pause, and occupancy barely moves, because the objects are still reachable and therefore not garbage.

# Turn on GC logging before you need it. On an already-running# process, jstat gives you the same signal live: watch OU (old used)# across full collections.jstat -gcutil 2847 5s 12 # The flags to have set in advance, on Java 9 and later.# -Xlog:gc*:file=/var/log/app/gc.log:time,uptime:filecount=5,filesize=32m 

[GC pause (G1 Humongous Allocation) 12847M->12834M(14336M), 4.2s] jvm_gc_collection_seconds_sum: 847s of 3600s  = 23% of wall clockjvm_gc_pause_seconds_max:      12.4s 

A full collection moving occupancy from 12847M to 12834M has reclaimed about 13 MB out of nearly 13 GB. The collector is not failing, it is succeeding at a task with almost nothing to do: everything it examined is still referenced. Spending 23 percent of wall clock in GC while freeing nothing is the point at which you stop tuning the collector and start looking for what holds the references.

What do you look for in the heap dump?#

The dominant retained types, and what roots them. A histogram usually names the leak in its first four lines.

# Prefer a live dump over waiting for the next crash. The kernel OOM# killer produces no dump, so an already-restarted process climbing# again is your best evidence source.jcmd 2847 GC.heap_dump /var/log/heapdumps/payment-gateway.hprof # Cheaper first look when a full dump is too large to move: the# histogram alone often identifies the retaining type.jcmd 2847 GC.class_histogram | head -20 

1. org.hibernate.internal.SessionImpl   8,412 inst   2.1 GB2. com.app.model.PaymentTransaction      4.2M inst   3.8 GB3. com.app.model.ReconciliationEntry     2.1M inst   1.9 GB4. byte[]                                            1.8 GB 

Read the shape rather than the absolute numbers. ORM types at the top, rooted in an open session, point at a persistence-context leak: entities loaded into a first-level cache that is never cleared. Static collections at the top point at an application-level cache with no eviction. Classloader and metaspace growth points at repeated redeployment or dynamic proxy generation. Thread-local maps point at pooled threads retaining per-request state after the request ends.

The distinction matters because each has a different fix, and the histogram is what separates them in one command. Here the retention is rooted in SessionImpl, which means the fix is in how the session is used, not in the collector, the heap size or the host.

How do you fix a Hibernate session cache leak?#

Bound the persistence context, and bound the query that fills it. Doing one without the other leaves the failure mode intact.

Hibernate's first-level cache holds every entity loaded through a session for the lifetime of that session. A batch loop that reads a large result set and never flushes or clears is therefore accumulating by design, not by accident. Every row processed stays pinned until the session closes, which for a long-running job is the end of the run.

// The bug: one session, one unbounded query, no reset. Every entity// the query loads stays in the L1 cache for the whole loop.@Scheduled(cron = "0 0 21 * * *")void runReconciliation() {  var txns = txnRepository.findAll();   // 4.2M rows  for (var t : txns) {    save(reconcile(t));    // no entityManager.flush() / clear()  }} 

// The fix: flush writes and clear the persistence context on an// interval, so retention is bounded by the batch size rather than// by the size of the result set.int count = 0;for (PaymentTransaction txn : txns) {  entryRepository.save(reconcile(txn));  if (++count % 500 == 0) {    entityManager.flush();   // push pending writes    entityManager.clear();   // release the L1 cache  }} 

Clearing on an interval bounds retention, but the unfiltered query is still materialising the whole table before the loop begins. Pagination or a date filter bounds the load itself, which is the more durable fix.

# Keep the evidence for next time. The kernel OOM killer writes no# dump, so without this a repeat incident starts cold again.-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/var/log/heapdumps/ # Let the JVM hit its own ceiling before the kernel does, so you get# an OutOfMemoryError and a dump instead of a silent SIGKILL.MemoryMax=12G      # systemd unit, below host RAM 

That last control is worth adopting generally on VMs. A cgroup limit under host RAM converts an unobservable kernel kill into an observable JVM failure with evidence attached, which is a strictly better incident.

What does a real JVM OOM kill on a VM look like?#

Sherlocks AI investigated one on payment-gateway at 03:14 IST on 28 March 2026. Prometheus paged on host memory above 95 percent on prod-vm-03, and the kernel had killed the JVM two minutes earlier at 03:12. The service was down for 8 minutes until systemd brought it back at 03:20, and 1,247 transactions failed in that window.

A traffic spike was ruled out first: heap growth was monotonic from 21:00 and uncorrelated with any request pattern, and prod-vm-01 and prod-vm-02 carried the same API traffic at 62 and 58 percent memory while prod-vm-03 sat at 98.7 percent. The differentiator was that prod-vm-03 also ran batch jobs. An infrastructure fault was ruled out from dmesg, which recorded a clean kill of PID 2847 with anon-rss of about 14.2 GB, accounting for nearly the entire host. An undersized heap was ruled out as the root cause because the process blew past -Xmx8g and kept climbing to 14.2 GB without ever levelling off, so a larger heap would only have delayed the kill.

GC logs then showed old generation moving from 12847M to 12834M across a major collection while the collector consumed 23 percent of wall clock, which established retention rather than pressure. The heap dump named it: 8,412 SessionImpl instances holding 2.1 GB, 4.2 million PaymentTransaction at 3.8 GB and 2.1 million ReconciliationEntry at 1.9 GB, all reachable from an open Hibernate session. The nightly reconciliation job ran findAll() over 4.2 million rows in a single session and never called flush() or clear(). What made it fatal was a change two weeks earlier that widened the scope from a 30-day window of roughly 200,000 rows to an unfiltered findAll(), about a twentyfold increase, turning a latent pattern into an OOM. The full writeup is in the JVM memory leak investigation.

Note the shape of the cause: a change two weeks old, invisible on the day it shipped, surfacing as a page in the middle of the night. That is the same structure as a disk filled by a dropped logrotate config, and it is why correlating with the most recent deploy fails on long-lived VMs. For the container-local version of this failure, where a cgroup limit rather than host memory is the ceiling and the container restarts rather than the host degrading, see Kubernetes OOMKilled. More worked examples are on the Linux VMs examples page, and the selection algorithm itself is documented in the Linux kernel OOM killer documentation.

JVM OOM kill FAQ#

Why is there no Java stack trace when my JVM is OOM killed? Because the kernel sent SIGKILL. The process is terminated from outside with no signal handler, no shutdown hook and no opportunity to log. An empty application log is itself the evidence that the kill came from the operating system.

How do I confirm the kernel OOM killer was responsible? Check dmesg -T or journalctl -k for an "Out of memory: Killed process" line. It names the victim, its PID, its anon-rss and its oom_score_adj. If that record exists, the kernel did it.

Will increasing -Xmx fix an OOM kill? Not when the growth is unbounded. A larger heap moves the crash later without preventing it. Raise -Xmx only when the curve plateaus above the current ceiling, which is a sizing problem rather than a leak.

Why did the process use more memory than -Xmx allows? Because -Xmx bounds the Java heap, not the process. Metaspace, thread stacks, code cache, direct byte buffers and allocator overhead sit outside it, so RSS routinely exceeds the configured heap by a wide margin.

How do I tell a memory leak from ordinary GC pressure? Watch what a major collection reclaims. Pressure means occupancy drops noticeably each cycle. A leak means the collector pays the full pause and occupancy barely moves, because the objects are still reachable.

Why did this only start failing recently if the code has been there for months? Leaks scale with the work processed. A scope change that multiplies rows per run, such as replacing a date filter with an unfiltered query, can turn a pattern that never mattered into one that exhausts the host in a single run.

How do I get a heap dump when the kernel gives no warning? Take one from the live process with jcmd GC.heap_dump while it is climbing rather than waiting for the crash. Set -XX:+HeapDumpOnOutOfMemoryError as well, and add a cgroup MemoryMax below host RAM so the JVM hits its own ceiling first and produces a dump.