Alert info
- Severity
- Critical
- Detected by
- Prometheus Node Exporter + Grafana
- Alert
- Host memory usage exceeded 95% on prod-vm-03
- Time
- 2026-03-28 03:14 IST
- Service
- payment-gateway (prod-vm-03)
Host memory on prod-vm-03 crossed 95% and the kernel OOM killer terminated the payment-gateway JVM at 03:12 IST. Investigate the cause.
Hibernate session cache leak in batch reconciliation job grew JVM heap from 4 GB to 14 GB over 6 hours. Linux OOM killer terminated the process, causing 8 minutes of downtime.
- JVM Heap
- 14.2 GB
- Downtime
- 8 min
- Failed Txns
- 1,247
Entities identified
Infrastructure
prod-vm-03
host (16 GB RAM)
Host VM
prod-vm-03.Linux OOM killer
kernel
Linux out-of-memory killer.
Service
reconciliation-job
nightly cron 21:00
Nightly reconciliation batch job.
Data
Hibernate L1 cache
SessionImpl
Hibernate first-level session cache.
PaymentTransaction
txn table
Payment transaction database table.
JVM heap
G1 old gen
JVM heap memory.
External
Prometheus + Grafana
Node Exporter / JMX
Prometheus monitoring system.
Gateway
payment-gateway
JVM (Java 17 / Spring Boot)
Payment gateway JVM process.
How they relate
The same relationships as text
- Prometheus + Grafana scrapes prod-vm-03
- Prometheus + Grafana scrapes prod-vm-03. Monitors host memory usage.
- prod-vm-03 hosts payment-gateway
- prod-vm-03 schedules reconciliation-job. Schedules the batch job.
- reconciliation-job findAll() PaymentTransaction. Queries transactions.
- reconciliation-job opens, no clear() Hibernate L1 cache. Opens a Hibernate session.
- Hibernate L1 cache retains entities JVM heap. Caches entities in memory.
- payment-gateway allocates JVM heap
- JVM heap exhausts host Linux OOM killer. Memory exhaustion triggers OOM killer.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Traffic spike drove the heap upRuled out
Compare request volume metrics against heap growth and check memory on peer VMs.
Heap growth is monotonic from 21:00 and is not correlated with any request pattern. prod-vm-01/02 carry the same API traffic and sit at 62% / 58%.
Only prod-vm-03 is in trouble; the two API-only peers are healthy.
host role memory status
prod-vm-01 API traffic 62% healthy
prod-vm-02 API traffic 58% healthy
prod-vm-03 API + batch jobs 98.7% OOM killed
If traffic were the cause, the peers would track too. They do not. The differentiator is the batch job.
Infrastructure fault or noisy neighbor on prod-vm-03Ruled out
Analyze system logs (dmesg) to identify the cause of the process termination.
dmesg shows a clean OOM kill of PID 2847 (java) with anon-rss 14.2 GB, accounting for nearly all of the 15 GiB host. No hardware fault.
The java process is the named victim and owns essentially all host memory.
[2026-03-28T03:12:04+0530] Out of memory: Killed process 2847 (java)
total-vm:18874368kB, anon-rss:14893056kB, file-rss:12288kB
oom_score_adj: 0
A single process exhausted memory. This is application behavior, not an infra fault or neighbor.
-Xmx8g heap is just sized too small for the workloadRuled out
Evaluate heap trajectory against configured limits over time.
The heap blew past -Xmx8g (8.9 GB by 01:00, incl. off-heap) and kept climbing to 14.2 GB. The growth is unbounded, so a bigger heap would only delay the OOM. Sizing is contributing, not the cause. Ruled out as root.
Heap is monotonic and crosses the configured limit without leveling off.
time heap note
21:00 4.1 GB baseline (batch starts)
23:00 6.3 GB climbing
01:00 8.9 GB past -Xmx8g (off-heap)
03:00 13.8 GB near host limit
03:12 14.2 GB OOM killed
No plateau. A raised -Xmx delays but does not prevent the crash; the real fault is unbounded retention.
Something in the JVM retains objects GC cannot reclaimConfirmed
Analyze GC logs and heap behavior for signs of memory retention.
The service is not erroring or slow, yet old gen barely moves across major GC (12847M to 12834M). The objects are live, not garbage. A leak holds references. Confirmed; find what holds them.
Major collections free almost nothing; collector spends 23% of the last hour running.
[GC pause (G1 Humongous Allocation) 12847M->12834M(14336M), 4.2s]
jvm_gc_collection_seconds_sum: 847s of 3600s = 23%
jvm_gc_pause_seconds_max: 12.4s
Old gen does not shrink, so the retained objects are reachable. This is a reference leak, not GC pressure.
The retained objects are Hibernate ORM entities, not a static cacheConfirmed
Analyze a heap dump histogram to identify the dominant retained object types.
The top retained objects are all ORM types rooted in SessionImpl: SessionImpl 2.1 GB, PaymentTransaction 3.8 GB, ReconciliationEntry 1.9 GB. No rogue static structures or metaspace growth. The session holds everything.
Largest retained sets are Hibernate session and the entities it loaded.
1. org.hibernate.internal.SessionImpl 8,412 inst 2.1 GB
2. com.app.model.PaymentTransaction 4.2M inst 3.8 GB
3. com.app.model.ReconciliationEntry 2.1M inst 1.9 GB
4. byte[] 1.8 GB
Everything large is reachable from the open Hibernate session's L1 cache. Not a static or classloader leak.
Batch reconciliation job loads 4.2M rows in one session without clear()Root cause
Correlate code execution paths with the onset of the memory climb.
The nightly job runs findAll() over 4.2M PaymentTransaction rows in a single Hibernate session and never calls flush()/clear(), so the L1 cache pins every entity for the full 6h12m loop. The scope was changed 2 weeks ago from a 30-day window (~200K rows) to findAll(), a ~20x jump that turned a latent pattern into an OOM.
Single unbounded query, loop with no session reset, recently widened scope.
@Scheduled(cron = "0 0 21 * * *")
void runReconciliation() {
var txns = txnRepository.findAll(); // 4.2M rows
for (var t : txns) {
save(reconcile(t));
// BUG: no entityManager.flush()/clear()
}
}
// 2 weeks ago: findByDateAfter(30d) ~200K -> findAll() 4.2M
All 4.2M entities stay in the L1 cache for the whole run. The scope change is the trigger that made it fatal.
Heap climb starts exactly at the 21:00 cron and ends in an 8-minute outage.
21:00 batch starts, heap 4.1 GB
03:12 heap 14.2 GB -> OOM kill (PID 2847)
03:12 systemd restart scheduled
03:20 service back up downtime ~8 min, 1,247 failed txns
The climb is locked to the batch schedule and the outage tracks the kill-to-restart gap. Confirms the job as root cause.
An application thread-local cache is unboundedRuled out
Check the heap dump for large thread-local maps.
No significant thread-local retention was found; the retention is rooted in the Hibernate session.
Final RCA
Batch reconciliation job loads 4.2M rows in one session without clear()
The nightly job runs findAll() over 4.2M PaymentTransaction rows in a single Hibernate session and never calls flush()/clear(), so the L1 cache pins every entity for the full 6h12m loop. The scope was changed 2 weeks ago from a 30-day window (~200K rows) to findAll(), a ~20x jump that turned a latent pattern into an OOM.
Remediation
# Immediate: Restart the service (already done by systemd)
sudo systemctl restart payment-gateway
# Fix: Add session.clear() in batch loop (every 500 records)
int count = 0;
for (PaymentTransaction txn : txns) {
ReconciliationEntry entry = reconcile(txn);
entryRepository.save(entry);
if (++count % 500 == 0) {
entityManager.flush();
entityManager.clear(); // Release L1 cache
}
}
# Or better: Use Spring @Modifying with pagination
# Process in batches of 1000 with Pageable- Add
entityManager.flush()andentityManager.clear()every 500 records in the batch reconciliation loop. - Revert to the 30-day date filter or implement pagination (Pageable) to limit loaded entities per iteration.
- Set JVM heap dump on OOM (-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=
/var/log/heapdumps/) for future analysis. - Add a Grafana alert for JVM heap growth rate: if heap grows > 1 GB/hour sustained, alert before OOM.
- Consider running batch jobs on a dedicated VM with higher memory, separate from API traffic serving.
- Add memory cgroup limits to the systemd service to trigger graceful shutdown before kernel OOM kills.
The batch job's failure mode was invisible until it was catastrophic because nothing measured the rate of heap growth, only the point of exhaustion. Pairing the flush/clear fix with a Grafana alert on sustained heap growth rate turns a six-hour silent climb into an early warning long before the kernel has to intervene. Running batch workloads on a dedicated host, separate from the two VMs serving live API traffic, also bounds the blast radius of the next unbounded query to one process instead of a shared pool, and a heap-dump-on-OOM flag means the next incident starts with evidence in hand rather than a cold restart.
Frequently asked questions
What actually killed the payment-gateway process?
A JVM process on the host was killed by the Linux kernel out-of-memory killer after its heap grew steadily for six hours, from 4.1 GB at 21:00 to 14.2 GB at 03:12, while host memory reached 98.7 percent. The kernel logged the kill directly in dmesg, with anon-rss of about 14.2 GB, naming the java process as the victim.
How did the investigation confirm a Hibernate cache leak rather than a generic memory leak?
A heap dump taken from the restarted process, together with GC logs showing old generation barely shrinking across major collections, showed the largest retained objects were org.hibernate.internal.SessionImpl and the entities it had loaded: 8,412 SessionImpl instances and millions of PaymentTransaction and ReconciliationEntry rows, all still reachable through the open session rather than eligible for collection.
Why did the Hibernate session keep growing instead of releasing memory?
The nightly batch reconciliation job called txnRepository.findAll(), loading every row into a single Hibernate session, then looped over the results without ever calling entityManager.flush() or entityManager.clear(). Every entity the query loaded stayed pinned in the Hibernate first-level cache for the full six-hour run, so the JVM heap grew continuously instead of releasing memory as rows were processed.
Why did this leak turn into an outage now instead of earlier?
Two weeks before the incident, the job scope was widened from a 30-day window (about 200,000 rows) to an unfiltered findAll() (4.2 million rows), roughly a 20x increase in entities retained per run. That change alone did not cause the leak, but it turned an existing pattern that had never mattered into one large enough to exhaust host memory in a single run.