Alert info
- Severity
- Critical
- Detected by
- Kubernetes / prod-india-exchange
- Alert
- Pod support-dashboard-staging-7cc9fd7874-gpf82 OOMKilled
- Time
- 2026-04-17 10:39:48 UTC
- Service
- support-dashboard-staging (prod-india-exchange)
The pod was terminated by the kernel OOM killer (Reason=OOMKilled, Exit 137) at 10:39:43Z and restarted. Investigate the cause.
A manual show_manual_review_kyc request processed 22.36M KycStatusLog rows in-memory, breaching the 2.5 GB container limit. The pod was OOMKilled (Exit 137) and readiness probes failed. Node memory sat at 3.12%, so the breach was purely container-local, not node pressure.
- Exit Code
- 137 (OOMKilled)
- Memory Limit
- 2,500 MB
- KycStatusLog
- 22.36M rows
Entities identified
Infrastructure
container cgroup
memory limit
Container memory limits configuration.
ip-10-50-6-210
worker node
Worker node in
ap-northeast-1.readiness probe
HTTP health
Health check for the pod.
Service
support-dashboard-staging
Django / gunicorn
Pod
support-dashboard-staging-7cc9fd7874-gpf82.show_manual_review_kyc
KYC review path
Manual KYC review code path.
Data
KycStatusLog
Postgres table
Postgres table for KYC status logs.
Identity
support agent
manual action
Internal support operator who initiated the workflow.
Gateway
Kubernetes
prod-india-exchange
Control plane / kubelet.
How they relate
The same relationships as text
- Kubernetes schedules / restarts support-dashboard-staging
- support agent invokes show_manual_review_kyc. Manual
show_manual_review_kycrequest. - support-dashboard-staging serves show_manual_review_kyc
- show_manual_review_kyc queries KycStatusLog. KYC review path pulls KycStatusLog.
- support-dashboard-staging bounded by container cgroup. Container memory is capped.
- container cgroup scheduled on ip-10-50-6-210. cgroup lives on the node.
- readiness probe probes support-dashboard-staging. Monitors the service health.
Hypotheses tree
Ruled-out branches are collapsed. Open any one to read the check that eliminated it.
Node-level memory pressure / noisy-neighbor evictionRuled out
Check the OOM semantics and node memory to determine if the node was under pressure.
Node was at 3.12% of 30,808 MB allocated and the kill carried Reason=OOMKilled / Exit 137, which is a cgroup limit breach, not an eviction.
Allocatable vs allocated on the worker hosting the pod.
Node: ip-10-50-6-210.ap-northeast-1.compute.internal
Allocatable memory: 30,808 MB
Allocated: 3.12%
Node mem [#--------------------] 3.12% ok
Vast node headroom. No node pressure could have driven this.
Termination reason and exit code distinguish OOM-kill from eviction.
OOMKilled / Exit 137 is a cgroup breach, not a node eviction. Ruled out.
Deployment / config regression introduced the OOMRuled out
Diff deploy history and configuration to identify recent changes.
No rollout at incident time; start command and memory settings unchanged (start_gunicorn.sh, 1500M/2500M). Favors a runtime workload trigger.
Start command and memory request/limit, with no recent image change.
command: /bin/bash start_gunicorn.sh --log-level=debug
requests.memory: 1500M
limits.memory: 2500M
recent rollout: none in the incident window
Nothing shipped to correlate. The trigger is runtime, not a release.
Gradual memory leak (recurring) rather than a one-shot spikeRuled out
Check the restart counts and sibling pods to determine if there is a recurring memory leak.
RESTARTS=1 (a single event), sibling pods at 0 restarts, and the pod recovered and stayed Running. Event-driven, not a leak.
Restart count on the affected pod vs its siblings.
NAME RESTARTS STATUS
support-dashboard-staging-...-gpf82 1 Running
support-dashboard-staging-...-r4k2x 0 Running
support-dashboard-staging-...-w9ld8 0 Running
Single restart, siblings clean. A leak would recur and span pods. Ruled out.
Container-local memory limit breach (cgroup), not node pressureConfirmed
Query the pod termination reason, exit code, and memory limits.
Last State Terminated, Reason=OOMKilled, Exit Code=137 at 10:39:43Z against the 2500M limit while the node sat at 3.12%. The kernel OOM killer fired on the cgroup. Confirmed; trace what filled it.
Last terminated state and exit code on the affected container.
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Finished At: 2026-04-17T10:39:43Z
Current State: Running
Restart Count: 1
A real OOM kill at the cgroup level. Exit 137 = SIGKILL from the OOM killer.
Container limit vs node allocation at the kill.
container limit 2500M <- breached
node allocated 3.12% ok
Memory [####################--] ~95%+ container limit
Node mem [#---------------------] 3.12% healthy
Breach is container-local against the 2500M cgroup, not node-level.
The manual show_manual_review_kyc request filled the budgetConfirmed
Review pre-crash logs to identify the workload executing prior to the OOM.
A manual show_manual_review_kyc request at 10:34:59Z logged 'Total count of KycStatusLog: 22,364,964' at 10:35:02Z, the only non-/health workload before the kill 4 minutes later. The KYC review path is the trigger.
Workload sequence from the manual request to the OOM.
10:32:52Z INFO /health -> 200
10:34:59Z INFO Manual: Request received for show_manual_review_kyc
10:35:02Z INFO Manual: Total count of KycStatusLog: 22,364,964
10:35:26Z INFO Manual: Query completed for show_manual_review_kyc
10:39:42Z INFO /health -> 200
10:39:43Z --- OOMKilled (Exit 137)
Only one real workload ran: the KYC review over a 22.36M-row table. Tight 5-min path to OOM.
Probe failures bracket the memory pressure and restart.
Warning Unhealthy Readiness probe failed: HTTP 502
Warning Unhealthy Readiness probe failed: timeout
Consequence of memory pressure and warmup, not an independent cause.
Unbounded query over 22.36M KycStatusLog rows breached 2.5 GBRoot cause
Analyze the memory footprint of the fetched KycStatusLog data.
KycStatusLog holds 22,364,964 rows. At ~100 bytes/row that is ~2.1 GB in-memory; with ~400 MB runtime overhead it consumes the entire 2500M budget. The unbounded KYC query materialized the full rowset and tripped the OOM killer. Root cause; repeatable until the path is paginated/streamed.
Row count translated to in-memory footprint vs the container limit.
KycStatusLog rows 22,364,964
~100 bytes/row ~= 2.1 GB in-memory
runtime overhead ~= 400 MB
total ~= 2.5 GB
container limit 2,500 MB <- exhausted
Full materialization of 22.36M rows alone exhausts the budget. Mechanism matches the OOM exactly.
No pagination, streaming, or .iterator() on the KycStatusLog read.
An unbounded read pulls all 22M rows into the app tier at once. This is the defect; raising the limit only delays it as the table grows.
Memory leak in the show_manual_review_kyc endpointRuled out
Monitor the endpoint memory usage over multiple requests.
Memory is garbage collected properly when small result sets are returned; the issue is the size of the single result set.
Background task consumed excessive memoryRuled out
Check the celery worker logs and task queues.
No background tasks were running on the pod at the time of the crash.
Final RCA
Unbounded query over 22.36M KycStatusLog rows breached 2.5 GB
KycStatusLog holds 22,364,964 rows. At ~100 bytes/row that is ~2.1 GB in-memory; with ~400 MB runtime overhead it consumes the entire 2500M budget. The unbounded KYC query materialized the full rowset and tripped the OOM killer. Root cause; repeatable until the path is paginated/streamed.
Remediation
# Confirm the OOM kill and current limits on the affected deployment
kubectl describe pod support-dashboard-staging-7cc9fd7874-gpf82 \
| grep -A5 'Last State'
# Inspect the KycStatusLog table size before re-running any review path
psql -c "SELECT reltuples::bigint AS approx_rows, \
pg_size_pretty(pg_total_relation_size('kycstatuslog')) AS total_size \
FROM pg_class WHERE relname = 'kycstatuslog';"
# Find other unbounded reads of the same model across the codebase
grep -rn 'KycStatusLog.objects' --include='*.py' \
| grep -v -E '\\.(filter|iterator|count|exists|only)\\('
# Watch container memory against the limit while validating the fix
kubectl top pod -l app=support-dashboard-staging --containers- Paginate the
show_manual_review_kycpath with an explicit page size (for example 500 rows) so no single request can materialize the full table. - Use
QuerySet.iterator()with a boundedchunk_sizefor any code path that must stream large result sets, avoiding full-rowset materialization in the worker. - Add a pre-query row-count guard that refuses or forces filtering when the estimated result set exceeds a safe threshold, returning a clear message to the support agent.
- Audit remaining admin and support endpoints for unbounded
.objects.all()reads against high-cardinality tables, and apply the same pagination pattern. - Add a container memory-usage alert at roughly 80% of the 2500M limit so pressure is visible before the OOM killer fires, rather than after the restart.
- Treat raising the memory limit as a stopgap only; the footprint scales with KycStatusLog growth, so the access pattern is the durable fix.
The defect is fully repeatable, since any support agent invoking show_manual_review_kyc materializes the entire KycStatusLog table again, and the table only grows over time, so raising the memory limit buys time rather than closing the gap. Pagination or a bounded iterator on that path removes the unbounded read at its source, and a pre-query row-count guard stops a future full-table read before it starts rather than after the container is already under pressure. A memory-usage alert around 80% of the cgroup limit gives an earlier signal than the kernel OOM killer, but it is a detection improvement, not a substitute for fixing the access pattern itself.
Frequently asked questions
How do you tell an OOMKilled container from a node-level eviction?
Check the termination reason and exit code against node memory headroom. In this incident the pod's Last State showed Reason=OOMKilled and Exit Code 137, which is the kernel OOM killer acting on a cgroup, while the worker node was allocated only 3.12% of 30,808 MB. A kubelet eviction would instead show Reason=Evicted with node-level pressure conditions present, neither of which appeared here.
Why did a single manual request cause an OOMKilled crash instead of a gradual memory leak?
Because the failure profile pointed at a one-shot event rather than accumulation. The affected pod showed RESTARTS=1 while two sibling replicas running identical code stayed at 0 restarts, and the pod recovered cleanly afterward. A real leak would recur across restarts and affect every replica running the same image; a single oversized query does neither.
How much memory does materializing 22 million database rows actually use?
In this incident, 22,364,964 KycStatusLog rows at roughly 100 bytes per row worked out to about 2.1 GB once fully loaded into the Python process, and adding around 400 MB of gunicorn and Django runtime overhead accounted for essentially the entire 2,500 MB container limit. The estimate was derived from the schema, not a measured heap profile, since no memory profiler was attached before the crash.
Does raising the container memory limit fix an unbounded query OOM?
Only temporarily. In this incident the root cause was an unpaginated read of KycStatusLog with no pagination, streaming, or .iterator() call, so the request pulls the entire table into memory regardless of the limit. Because the table keeps growing, raising the limit only postpones the next OOM kill; the durable fix is bounding the query itself.