How to fix a Kubernetes OOMKilled container (exit code 137)?

If you have seen a pod restart with Reason: OOMKilled and Exit Code: 137, the container asked for more memory than it was allowed and the kernel ended it mid-run. The number 137 is not random. It is 128 plus 9, and signal 9 is SIGKILL, the kernel forcibly stopping a process. So exit 137 reads as: this process was killed, and in Kubernetes the usual reason is memory.

What does OOMKilled mean in Kubernetes?#

OOM stands for out of memory. When a container exceeds the memory limit set on it, the kernel's OOM killer steps in and terminates the process inside that container. Kubernetes records it as OOMKilled and the container exits with code 137. The concept comes straight from the Linux kernel, described in the out-of-memory behavior that governs any process, containerized or not.

The important thing to understand up front is that this is a limit being hit, not necessarily a machine running dry. Each container gets a memory ceiling through a cgroup, and the kernel enforces it per container. A container can be OOMKilled on a node with plenty of free memory, simply because that one container went over its own allocation.

Is OOMKilled a container problem or a node problem?#

This is the first fork, and getting it right decides where the fix belongs.

Container-local OOM is the common case. The container exceeded its own memory limit while the node was fine. The fix lives in the workload: its limit, its code, or its behavior.

Node-level memory pressure is different. The whole node runs low on memory, and the kubelet starts evicting pods to protect it, which is a separate mechanism documented under node-pressure eviction. That shows up as eviction, not as an OOMKilled exit 137 on a single container.

The way you tell them apart is simple: check the node's memory. If the node had headroom and one container still died, it was container-local, and no amount of adding nodes will help. You have to fix the workload.

What causes a Kubernetes OOMKilled?#

A few patterns cover almost every case.

The memory limit is set too low. The most common cause is the simplest. The limit was set conservatively, or copied from another service, and it is just lower than what this workload actually needs under real load. The container was never given enough room.

A single unbounded operation. One request loads far more into memory than expected, all at once. A query with no pagination that pulls millions of rows, a large file read fully into memory, a bulk job that builds an enormous in-memory structure. The service runs fine for weeks, then one heavy operation blows straight through the limit.

A memory leak. The application slowly accumulates memory it never releases. Here the pattern is telling: memory climbs steadily over hours or days, gets OOMKilled, restarts clean, and climbs again. A sawtooth on the memory graph is the signature of a leak.

A load spike. More concurrent work than the limit was provisioned for. Each request is reasonable, but the sum during a traffic burst exceeds the ceiling.

Runtime heap not aligned with the container limit. For the JVM, Go, Node, and others, if the runtime thinks it has more memory than the container limit allows, it will happily grow past the limit and get killed. The runtime's memory settings have to be aware of the container's real ceiling.

How do you confirm it was OOMKilled?#

Go straight to the pod and read its last state.

kubectl describe pod <pod-name> -n <namespace> 

In the output, under the container's Last State, you are looking for:

Last State:   Terminated  Reason:     OOMKilled  Exit Code:  137 

That confirms the kernel killed it for memory. Next, check what limit it was working against:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.limits}' 

Then confirm whether the node was actually under pressure, to rule out node-level memory issues:

kubectl describe node <node-name> 

Read the Conditions for MemoryPressure. If it says False, the node was fine and the OOMKill was purely container-local, which points the fix at the workload, not the cluster.

To find every container the kernel has killed rather than the one you were paged about:

# Every OOMKilled container across all namespaces, with its restart countkubectl get pods -A -o json | jq -r '  .items[] as $p | $p.status.containerStatuses[]?  | select(.lastState.terminated.reason == "OOMKilled")  | "\($p.metadata.namespace)/\($p.metadata.name)\t\(.name)\trestarts=\(.restartCount)"' 

Then compare what the container actually uses against what it is allowed:

# Live usage, per container rather than per podkubectl top pod <pod-name> -n <namespace> --containers 

kubectl top is a snapshot, and an OOMKill is a peak, so a container sitting comfortably now tells you nothing about the moment it died. For the peak you need the series, which Prometheus exposes directly:

# Working set as a fraction of the limit, worst containers firsttopk(10,  max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})  /  max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})) 

Anything sustained above roughly 0.9 is a future OOMKill with a date on it. The shape of that series is also the diagnosis: a vertical spike is an unbounded operation, a slow climb with a reset at every restart is a leak.

How do you fix an OOMKilled pod?#

The fix depends on which cause you confirmed, so match it rather than reflexively raising the limit.

If the limit was simply too low, right-size it. Set the memory limit to match real usage plus sensible headroom:

resources:  requests:    memory: 512Mi  limits:    memory: 1Gi 

If a single operation is the trigger, bound it. This is the durable fix when one request loads too much at once. Paginate the query, stream the file instead of reading it whole, or process the batch in chunks so peak memory stays flat regardless of input size. Raising the limit here only delays the next OOMKill until the input grows again.

If it is a leak, the limit is not the problem and raising it just makes the crashes less frequent, not gone. Profile the application, find what is not being released, and fix it in the code. The limit is a safety net, not a cure for a leak.

If the runtime heap is the issue, align it to the container. A managed runtime that sizes its heap from the host rather than the cgroup will grow straight past the limit and be killed for it, so the ceiling has to be expressed in the runtime's own terms:

env:  # JVM: derive the heap from the container limit, not the node's RAM  - name: JAVA_TOOL_OPTIONS    value: "-XX:MaxRAMPercentage=75.0"  # Node.js: cap old-space below the limit (here, ~75% of a 2Gi limit)  - name: NODE_OPTIONS    value: "--max-old-space-size=1536" 

Percentages beat fixed sizes here, because a hardcoded -Xmx silently becomes wrong the moment someone edits the limit, and the two drift apart with nothing to catch it.

How do you prevent OOMKills?#

Set requests and limits from real data, not guesses. Measure actual memory usage under load and set the limit above the real peak with headroom, so normal spikes do not trip it.

Then watch the trend. Alert on memory usage approaching the limit, not just on the OOMKill after the fact. A container sitting at 90 percent of its limit is a warning you can act on before the kernel acts for you. And bound the operations that can balloon: pagination and streaming turn an unpredictable memory profile into a flat one.

What does a real OOMKilled incident look like?#

Sherlocks AI investigated one end to end. A container on a support dashboard was OOMKilled with exit code 137, and readiness probes started failing. The trigger was a single manual request, a show_manual_review_kyc operation that processed 22.36 million KycStatusLog rows in memory at once, which breached the container's 2.5 GB limit.

The signal that made the diagnosis clean: node memory sat at just 3.12 percent. The node had enormous headroom. So this was not node pressure at all, it was purely container-local, one unbounded query loading millions of rows into a 2.5 GB container. That single fact moved the fix from the cluster to the workload, specifically to bounding that query. The full writeup, including the paths that were ruled out, is in the OOMKilled KYC query investigation.

It is a textbook example of the unbounded-operation cause, and a reminder that the OOMKill is the symptom. The real question is always what consumed the memory and why the workload let it. A closely related storage version of the same theme is covered in our pod eviction guide, and more real investigations are on the Kubernetes examples page. On managed platforms like EKS the same container-limit rules apply, as noted in the Amazon EKS documentation, and correlating memory against deploys and events, ideally through open standards like OpenTelemetry, is what separates a spike from a leak.

Kubernetes OOMKilled FAQ#

What does exit code 137 mean? The container was killed by SIGKILL, which in Kubernetes almost always means it exceeded its memory limit and was OOMKilled.

Is OOMKilled the application's fault or the node's? Usually the container's. If the node had free memory and one container still died, it exceeded its own limit.

Does raising the memory limit fix OOMKilled? Only if the limit was genuinely too low. For a leak or an unbounded query, a higher limit just delays the next kill.

How do I know if it is a memory leak? Memory that climbs steadily, gets killed, restarts clean, and climbs again. A sawtooth graph is the tell.

What is the difference between OOMKilled and an eviction? OOMKilled is one container exceeding its own memory limit. Eviction is the node running low on a resource and the kubelet removing pods.

Why did readiness probes fail during the OOMKill? The process was killed mid-run, so it stopped answering health checks until it restarted.