RCA guide

How to fix a Kubernetes CrashLoopBackOff?

Sherlocks AIKubernetes9 sections10 Min Read
kubectl — production cluster
sre@prod-bastion ~ $ kubectl describe pod ad-serviceStatus:      CrashLoopBackOff  (7 restarts, age 21m)Last State:  Terminated  Exit Code: 1Events:      Back-off restarting failed container# The state is the loop, not the reason. The reason is one# command away, in the output of the container that died. sre@prod-bastion ~ $ kubectl logs ad-service-767c6f45c-xgvkb --previous

If a pod is stuck in CrashLoopBackOff, one container inside it starts, exits, gets restarted, exits again, and Kubernetes keeps stretching the delay between restarts. The name describes the loop, not the reason. Four pods can all show CrashLoopBackOff and have four completely unrelated causes underneath. So the job is never to fix CrashLoopBackOff. It is to find out why the container keeps dying.

What does CrashLoopBackOff mean?

CrashLoopBackOff is a status the kubelet reports when a container has failed repeatedly and is being restarted with an increasing delay. The BackOff part is the important detail. After each crash, Kubernetes waits longer before trying again, doubling the delay up to a cap of five minutes, so it does not hammer a broken container in a tight loop.

That backoff behavior is also a diagnostic signal. If the delay between restarts is growing, the container is genuinely failing over and over. How long the container survives before each crash tells you a lot: a container that dies instantly is failing at startup, while one that runs for a while and then crashes is failing during operation. Those are different problems.

How do you read the logs of a container that already crashed?

This is the single most useful command for CrashLoopBackOff, because the current container may be too young to hold the evidence. You want the output of the instance that already died:

bash
kubectl logs <pod-name> -n <namespace> --previous 

The --previous flag shows the logs from the last terminated container, not the one currently trying to start. That is where the actual error usually is: the stack trace, the missing environment variable, the connection refused. Without --previous, you are reading a container that has not failed yet, which tells you nothing.

Pair it with a describe to see the state and the exit code:

bash
kubectl describe pod <pod-name> -n <namespace> 

Under the container's Last State you will see Terminated, a Reason, and an Exit Code. Those two commands, previous logs and the exit code, resolve most CrashLoopBackOff cases.

If you would rather not read a full describe, the same three fields come straight out of the API, one line per container:

bash
# Reason, exit code and restart count for every container in the podkubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\trestarts="}{.restartCount}{"\n"}{end}' 

That matters on a multi-container pod, where a describe buries which of the containers is actually the one looping.

To see the whole blast radius rather than the one pod you were paged about:

bash
# Every container currently in a backoff loop, across all namespaceskubectl get pods -A -o json | jq -r '  .items[] as $p | $p.status.containerStatuses[]?  | select(.state.waiting.reason == "CrashLoopBackOff")  | "\($p.metadata.namespace)/\($p.metadata.name)\t\(.name)\trestarts=\(.restartCount)"' 

One pod looping is a workload problem. Twenty pods across unrelated services looping at the same moment is a cluster or dependency problem, and that distinction changes where you look next.

The backoff timing itself is in the events, which is how you tell a container that dies instantly from one that runs a while first:

bash
kubectl get events -n <namespace> \  --field-selector involvedObject.name=<pod-name> --sort-by=.lastTimestamp 

Widening gaps between Back-off restarting failed container events confirm the loop is real rather than a single bad start.

What do the exit codes mean?

The exit code narrows the cause fast. Three come up constantly.

Exit code 1. A generic application error. The process started, hit something it could not handle, and exited. This is the code that sends you straight to the previous logs, because the cause is almost always printed there: an unhandled exception, a failed config parse, a missing file. Exit 1 means read the output.

Exit code 137. The container was killed by SIGKILL, and in Kubernetes that almost always means it was OOMKilled for exceeding its memory limit. This is a resource problem, not a logic problem, and it has its own investigation path. If you see 137, work it as an out-of-memory issue, covered in detail in our OOMKilled guide.

Exit code 143. The container received SIGTERM and shut down. This is often a graceful termination, so a lone 143 is not always a crash. But if it shows up in a restart loop, something is repeatedly asking the container to stop, a failing liveness probe, a resource limit, or an orchestration event. Look at what is sending the signal.

Codes above 128 map to signals: subtract 128 and you get the signal number. 137 is 128 plus 9 (SIGKILL), 143 is 128 plus 15 (SIGTERM). That single piece of arithmetic tells you the container was stopped by the system rather than exiting on its own.

What causes CrashLoopBackOff?

With the exit code and previous logs in hand, the cause usually falls into one of these.

A failed dependency at startup. The container cannot reach something it needs to boot, a database, a message broker, a cache, an internal service. It tries, fails, exits, and loops. The tell is in the logs: connection refused, timeout, name resolution failure. The container's own code is fine. Something it depends on is not there.

A fault in the application code. The process itself is broken. A bad configuration value, a failed assertion, a missing required environment variable, an exception on startup. Here the container would fail even if every dependency were healthy. The logs show a stack trace or an error from inside the application, not a network error reaching outward.

A bad configuration or missing secret. A required environment variable or config file is absent or wrong, so the app refuses to start. Often visible as a clear startup error naming the missing key.

An out-of-memory kill. Exit 137. The container exceeds its memory limit and gets killed, restarts, and does it again. Resource issue, not a logic one.

A failing liveness probe. The container starts fine but a misconfigured liveness probe declares it unhealthy and Kubernetes restarts it, repeatedly. Here the app is not crashing at all, the probe is killing it. Check whether the probe's endpoint, timing, and thresholds actually match how the app behaves on startup.

How do you tell a failed dependency from a code fault?

This is the fork that decides where the fix goes, and it is the one people get wrong most often, because both look identical from the outside: the same CrashLoopBackOff, the same restart loop.

The previous logs settle it. A dependency failure points outward: connection refused to a host, a timeout reaching a service, DNS resolution failing. The container is healthy and cannot reach something. A code fault points inward: a stack trace from the application, a config parse error, an assertion failure. The container is broken regardless of what it can reach.

The distinction matters because the fixes are opposite. A dependency failure is fixed by making the dependency reachable, or by making the container wait and retry instead of crashing when the dependency is briefly unavailable. A code fault is fixed in the application or its config. Treating one as the other wastes the whole investigation.

One nuance worth knowing: a dependency that fails and then recovers can leave a container that already crashed out and self-healed on the next restart, which makes the incident look resolved without an explanation. If the logs are gone by the time you look, the cause can vanish with them, which is exactly why capturing the previous output early matters.

How do you fix CrashLoopBackOff?

Match the fix to what the exit code and logs told you.

  • Failed dependency: make the dependency reachable, and add startup retry or a readiness gate so a brief outage does not crash the container outright.
  • Code fault: fix the bug or the config in the application, redeploy.
  • Missing config or secret: create or correct the environment variable, ConfigMap, or Secret the container needs.
  • Exit 137, OOM: right-size the memory limit or bound what is consuming memory. See the OOMKilled guide.
  • Failing liveness probe: align the probe's endpoint, initial delay, and thresholds with how the app actually starts, so a slow boot is not mistaken for a crash. The Kubernetes probe documentation covers the timing settings.

For that last one, a startupProbe is the fix people miss. It suspends the liveness probe until the app has actually booted, so a slow start cannot be mistaken for a wedged process:

yaml
# Up to 30 x 5s = 150s to boot, without loosening liveness afterwardsstartupProbe:  httpGet: { path: /healthz, port: 8080 }  periodSeconds: 5  failureThreshold: 30 livenessProbe:  httpGet: { path: /healthz, port: 8080 }  periodSeconds: 10  failureThreshold: 3 

Without the startup probe the usual workaround is a large initialDelaySeconds on liveness, which buys boot time at the cost of leaving a genuinely wedged container unrestarted for just as long. The startup probe separates the two questions, so neither answer has to be a compromise.

How do you prevent CrashLoopBackOff?

Make startup resilient to the things that are briefly unavailable. Dependencies blip, so a container that retries a database connection for a few seconds beats one that crashes the instant the database is not there. Get liveness and readiness probes right, use readiness to hold traffic during a slow boot and reserve liveness for a genuinely wedged process, so probes do not become the thing that kills a healthy container. And validate configuration and required secrets early, ideally failing with a clear message rather than a cryptic loop.

What does a real CrashLoopBackOff look like?

Sherlocks AI has investigated several, and the striking thing is how little they have in common. One was a Celery consumer that entered CrashLoopBackOff at startup because it could not establish a connection to its AMQP message broker, a failed dependency, which then self-healed on a later restart before anyone could explain it. Another, on an ad service, was a fatal mutex assertion inside the runtime triggered by a specific deployment revision, a genuine code fault that crashed the pod seven times before it was caught. Others traced to cluster resource contention and to a missing JVM argument that a connector required at startup.

Same symptom, four unrelated causes: a broker, application code, resource pressure, and a missing runtime flag. That is the whole point of CrashLoopBackOff. The state is identical every time and the cause never is. You can read these worked investigations, including the paths that were ruled out, on the Kubernetes examples page. Correlating the crash against deploy history and events, ideally through open standards like OpenTelemetry, is often what separates a code fault from a dependency blip. The underlying restart behavior is documented by Kubernetes, and on managed platforms like EKS the same lifecycle applies, per the Amazon EKS documentation.

Kubernetes CrashLoopBackOff FAQ

What does CrashLoopBackOff actually mean? A container keeps starting and crashing, so Kubernetes waits progressively longer between restarts.

How do I see why the container crashed? Run kubectl logs with the --previous flag to read the output of the container that already died.

What does exit code 1 mean? A generic application error. The cause is almost always in the previous logs.

What does exit code 137 mean in a crash loop? The container was OOMKilled for exceeding its memory limit. Work it as an out-of-memory issue.

What does exit code 143 mean? The container received SIGTERM. Often graceful, but in a loop something is repeatedly telling it to stop.

How do I know if it is my code or a dependency? The previous logs. Outward errors like connection refused mean a dependency. Inward errors like a stack trace mean the code.

See Sherlocks AI in action

Watch an AI SRE work a real incident from alert to root cause, on your stack, in 30 minutes.